Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[PeridexisErrant]

There's been a lot of talk lately about ways to make combining small mods easy for new players.  To me that means a GUI or GUIs, and given the proliferation of such things around here a top priority is a sensible standard way to package mods. 

Here's one of many mod manager discussions.  TLDR is that everyone likes the idea of building up something like a starter pack for mods, where new people can get their toes wet in a GUI that can stack mods.  The plot is basically to get a launcher, probably the PyLNP, and then take over the world start including optional and easy mods in everything. 

Regarding format, I think we need to nail this down soon.  The longer we leave it, the more likely it becomes that the community ends up with multiple patch formats... So my proposal is: 

- Each distinct mod gets it's own folder, named whatever the mod is called. 
 - In that folder, include the raw folder structure with any files changed from the vanilla raws; plus at the top level other stuff (eg readme, random files and folders) that can be ignored without error. 
 - Possibly also a manifest file if this takes off and needs extending to more arcane cases. 
 - One of the mod folders is the full vanilla raws; that folder is named identically to the version it came as, eg "df_40_08" (omit OS as it's irrelevant to raws).  Folder names starting "df_*" are reserved, case-insensitive. 
 - A *real* patch is derived when needed by running a diff between the latest vanilla folder (autodetected) and the mod
 - A folder "raw" is created (another reserved name; the old is deleted if present) and the vanilla raws copied in. 
 - Diffs are then applied to this folder in the specified order
 - If there's a conflict, highlight the problematic mod in the list and delete the raw folder (no invalid mods created here!).  This ensures compatibility and acts as a live preview for feedback.

 - Mods with modules, dependencies, dfhack, etc: all tricky.  I propose ignoring that for now; complex behaviour and handling comes after the basics are working and there's some interest from both modders and users.  Suggestions welcome anyway.

Advantages:
 - such a basic format makes repacking easy; if someone distributes a preinstalled mod all you have to do is remove the other files (easy for a launcher to do)
 - inversely, it's easy to install such a packaged mod without tools just by drag-and-dropping it over a vanilla install
 - a Mod Starter Pack becomes possible, and easily user-extensible (the latter is the coolest part)

I'd love to hear feedback on this idea. 

For more info, see https://github.com/PeridexisErrant/Py-Mod-Loader

[Dirst]

I really like this idea, and it brilliantly handles the idea of removing a mod: you just rebuild from vanilla skipping the mod-pack you no longer want.

The sequence you described would be a great starting point, but collision handling needs to happen sooner rather than later because virtually ALL custom buildings and reactions need to go into entity_default.txt.  I think the responsibility for treading carefully in vanilla files should be placed on the modder, with some handholding in the launcher's documentation.

For example, it is more reasonable to tell the modder to translate his/her positioning hints into regular expressions than it is to ask the user (who may not know the first thing about raw tokens) if a partial match is valid.  I do not recommend expecting the launcher to pile on special handling for umpteen different situations (inserting custom buildings, changing gaits, adjust stone tile, etc etc etc etc); pick one or two general formats.

[PeridexisErrant]

I really like this idea, and it brilliantly handles the idea of removing a mod: you just rebuild from vanilla skipping the mod-pack you no longer want.

Exactly.  As well as the difficulty of removing a patch, it also avoids breaking things by removing a dependency for something else.  This way, every combination the user tries is explicitly checked and rejected if it won't work. 

The sequence you described would be a great starting point, but collision handling needs to happen sooner rather than later because virtually ALL custom buildings and reactions need to go into entity_default.txt.  I think the responsibility for treading carefully in vanilla files should be placed on the modder, with some handholding in the launcher's documentation.

For example, it is more reasonable to tell the modder to translate his/her positioning hints into regular expressions than it is to ask the user (who may not know the first thing about raw tokens) if a partial match is valid.  I do not recommend expecting the launcher to pile on special handling for umpteen different situations (inserting custom buildings, changing gaits, adjust stone tile, etc etc etc etc); pick one or two general formats.

Yeah, more intelligent handling of collisions is going to be required before too long; I absolutely agree that special cases should be kept to a minimum - or preferably avoided entirely.  To avoid incompatibility there may eventually be a canonical way of doing it, but for now keep everything as simple as possible so it actually happens!

That's why I don't think 'soon' is now - if the first version usually only allows one mod to substantially alter entity_default.txt, that's OK for the first version. 

-----------------------------------------------------

And an idea on how to handle mods that have been split up into modules:

Note that this is not a final idea, and for now each module can simply be treated as a standalone mod with an indicative name.  This is much easier to handle, and probably more elegant overall.  Despite the annoyance for modders, it might just be best to require all parts of split mods to be standalone and treat them as such.  Conflicts or dependencies should be caught by the launcher anyway.

 - Single folder for the whole mod
 - subfolders for each part of it, each of which would be valid (if useless/conflicting) as a standalone mod
 - some standard way of declaring what should be loaded by default, and in what order
 - user can treat the collection as a single mod, or choose to see the more detailed settings

I think that this will require a manifest, which should not be modifiable (except obviously with a text editor).  The user can adjust the settings, but this is only held until the launcher is closed.  I'm leaning towards xml format; the second part of the manifest being the load order for submodules.  Given that there is no top-level "raw" folder, a missing manifest will mean no modules are loaded and the collection is treated as an empty mod.  The manifest can also include metadata such as name of mod, author, link to source, and description (so non-collection mods may also want one, though it's optional). 

Hopefully, mods which have graphics options can put the main mod in a top-level "raw" folder - ASCII raws, graphics turned off - which will always be loaded, and then have modules which add in the graphics.  This would actually handle a general case of options with shared dependencies.  Thinking this through raises some issues about diffs; for consistency these mod--specific graphics should be diffed from vanilla but I can see that getting impractical or conflicted. 

All of this more complex handling comes later though; for now a simple launcher that can do diffs and simply rejects conflicts will be a big step up.  First the simple, then the practical, then the possible, and last of all take over the world!

-----------------------------------------------------

Currently my biggest wish for mod-manager is some sort of state tracking. I.e. I don't like how it detects if mod is already merged in.

Second in wish list is this. Namely some smarter way of adding-removing everything and being raw aware. Imho there is no silver bullet: even multi-million programming industry did not crack this problem and there are so called "merge conflicts" that need to be sorted out by hand.

The rebuild-without-thing-to-remove approach handles this nicely, but only if the original set of mods and merge order is saved.  The simplest way to do this would be to create a text file in the built raw folder; "included_mods.txt".  First line includes a timestamp and ID for the launcher that created it, so people can detect if it was someone else's launcher.  Second line is the base DF raws, eg "df_40_08".  Subsequent lines give mods in order of application:  "first_mod_applied", "why do people put spaces in folder names", "thismodlast". 

I think that this is premature for this idea though, when we don't even have the basic concept implemented. 

[thistleknot]

Although this is a crosspost.

I was thinking of a class definition for tagToken's

And keeping track of tagToken's within entities would be a good start.

Spoiler: tagToken.h (click to show/hide)

Code: [Select]

#ifndef TAGTOKEN_H
#define TAGTOKEN_H

#include <QtCore>

class tagToken
{
public:
    tagToken();

private:
    QString nameOftagToken;

    //line # of file token was on
    int rowNumber = 0;

    //colNumber token started at
    int colNumber = 0;

    //Either an identifier such as Object:
    //or What is specified after Object: such as creature
    QString objectInstance ="";

    //is tagToken "Object:" ?
    bool objectIdentifier = 0;

    struct tokenInfo
    {
        bool openBrace = 0;
        bool closingBrace = 0;
        bool whiteSpace = 0;

        //it's a comment if there is no nearby [ or ]
        bool comment = 0;

        //are there newLines in this token?
        bool newLines = 0;
    };
};

#endif // TAGTOKEN_H


[Dirst]

Yeah, more intelligent handling of collisions is going to be required before too long; I absolutely agree that special cases should be kept to a minimum - or preferably avoided entirely.  To avoid incompatibility there may eventually be a canonical way of doing it, but for now keep everything as simple as possible so it actually happens!

That's why I don't think 'soon' is now - if the first version usually only allows one mod to substantially alter entity_default.txt, that's OK for the first version. 

Agreed, let's get something out the door.  The world we be there for conquering tomorrow.

And an idea on how to handle mods that have been split up into modules:

Note that this is not a final idea, and for now each module can simply be treated as a standalone mod with an indicative name.  This is much easier to handle, and probably more elegant overall.  Despite the annoyance for modders, it might just be best to require all parts of split mods to be standalone and treat them as such.  Conflicts or dependencies should be caught by the launcher anyway.

 - Single folder for the whole mod
 - subfolders for each part of it, each of which would be valid (if useless/conflicting) as a standalone mod
 - some standard way of declaring what should be loaded by default, and in what order
 - user can treat the collection as a single mod, or choose to see the more detailed settings

I think that this will require a manifest, which should not be modifiable (except obviously with a text editor).  The user can adjust the settings, but this is only held until the launcher is closed.  I'm leaning towards xml format; the second part of the manifest being the load order for submodules.  Given that there is no top-level "raw" folder, a missing manifest will mean no modules are loaded and the collection is treated as an empty mod.  The manifest can also include metadata such as name of mod, author, link to source, and description (so non-collection mods may also want one, though it's optional). 

This works, but there needs to be at least one of two failsafes, preferably both: (1) a way to declare dependencies (even if it's implemented as just a pretty-please note to the user) so that the diff can be made from the parent module, and/or (2) a way to gracefully handle attempts to diff things that just aren't there (non-fatal notification to the user, post v1.0 some kind of error handling as well).

Hopefully, mods which have graphics options can put the main mod in a top-level "raw" folder - ASCII raws, graphics turned off - which will always be loaded, and then have modules which add in the graphics.  This would actually handle a general case of options with shared dependencies.  Thinking this through raises some issues about diffs; for consistency these mod--specific graphics should be diffed from vanilla but I can see that getting impractical or conflicted. 

See above.  I think it's perfectly fine for the game to vomit into the errorlog if you load the TWBT widgets for workshops you aren't using resulting in a bunch of unknown tags.  Explicit dependencies would help prevent that kind of thing.

All of this more complex handling comes later though; for now a simple launcher that can do diffs and simply rejects conflicts will be a big step up.  First the simple, then the practical, then the possible, and last of all take over the world!

Maybe a placeholder of just reading the manifest file and printing a notice that it was noticed.  Hook into that later for actual logic.

Currently my biggest wish for mod-manager is some sort of state tracking. I.e. I don't like how it detects if mod is already merged in.

Second in wish list is this. Namely some smarter way of adding-removing everything and being raw aware. Imho there is no silver bullet: even multi-million programming industry did not crack this problem and there are so called "merge conflicts" that need to be sorted out by hand.

The rebuild-without-thing-to-remove approach handles this nicely, but only if the original set of mods and merge order is saved.  The simplest way to do this would be to create a text file in the built raw folder; "included_mods.txt".  First line includes a timestamp and ID for the launcher that created it, so people can detect if it was someone else's launcher.  Second line is the base DF raws, eg "df_40_08".  Subsequent lines give mods in order of application:  "first_mod_applied", "why do people put spaces in folder names", "thismodlast". 

I think that this is premature for this idea though, when we don't even have the basic concept implemented.

Again, I think it's worth a placeholder here... generate the state/order file when building the raws as a debug tool, even if the tool itself has no idea how to parse it yet.

Does this sound like a reasonable starting point?

[thistleknot]

note: I use tag/token interchangeably.

it seems like your whole idea is centered around the same concept I gleaned from using github.  Start with vanilla, then add to it and derive patches (really no different than software merging I guess, which is still something I just recently learned).  Basically, make whatever mod you want.

I think Putnam wanted to make the system for "packaging" the mods with patch files in the same manner.

That's what got me started with this mod tool merge idea.  Having to MANUALLY resolve conflicts is what I was thinking google-diff-patch-match might do 'auto-magically'.  I initially wanted to create a tool that patched raw files using that library, but... alas I decided to manually parse the files and keep track of tokens instead.  I would still like to implement the patch algorithm, but I think keeping track of an object's entire set of tags in their ordinal positions before and after merge should be enough to keep some sort of state tracking system for the object.

Anyways, when I realized why diff patch's conflicted, I figured if there was a special way we can work with the raws maybe we can isolate those merge conflicts, that's where my concept of tag tracking came in.  Keeping track of an entire object's tag state from both sets of files to be compared with (say base vanilla against mylittleponymod).  One can either see entire new entities created, or entities modded, or deleted, etc.  But it would be like a tag definition of the object before and after merge.  Like an additive/subtractive tag diff.

[PeridexisErrant]

@Dirst:  Yep, sounds like we're on the same page. 

How's this for a set of goals?
v0.1 - basic logic in place, folders findable, etc.  Written in Python for PyLNP compatibility
       - can derive a diff correctly and create a set of raws with one mod-patch applied
v0.2 - installs multiple non-conflicting mods correctly
       - handles errors and conflicts gracefully
       - write log file with mod merge order etc
       - place holder handling for a manifest file (note existence)
v0.9 - GUI time!  Integrated as a tab in the PyLNP - get it functional in new context, refactor to fit, etc.
    .1 - implement manifest and use information from it in display (if present)
       - 'simplify mod folders' option ala LNP, deleting extra files (not readme etc, but eg rest of DF install)
v1.0 - start finding or soliciting or formatting some mods

In roughly that order.  1.x probably includes storing patches instead of changed files and maybe declaring the version of the raws to diff against on a per-mod basis in the manifest.  Smart handling for mods with dependencies and a way to intelligently resolve merge conflicts without user input would take us to 2.0, but we can discuss that kind of thing in more detail once we get to 0.9!

@Thistleknot: Yeah, it's a pretty similar concept to your patch ideas and not by coincidence

Putnam inspired the format by pointing out that a standard format is, duh, the vanilla raws.  This has the important advantage over distributing just diffs that it's a standard format and would allow you to just dump a preinstalled mod, DF executable and all, into the folder and have it work. 

Honestly I'm hoping that by the time this gets up to building in a smarter way of handling merges all that needs to happen is we port you logic over.  Obviously that would be much easier if you took inputs in the same format as I'm proposing 

[Sean Mirrsen]

ModBase used to use something similar, it provided its own markup for tag adding and removal, and kept a listing of mods applied sequentially to each other, each with their own folders.

The big problem with it, as-is, is that it doesn't handle logical ordered clusters of tags yet. The introduction of the caste system way back when broke it, and I sort of left it there when I couldn't make it work. It needs a complete logic rewrite to handle being able to read and write tags to and from specific castes, specific attacks, tissue defs, etc.

It also needs a complete rewrite because it's all in VB.NET, but that's another thing.

[PeridexisErrant]

The thing to remember is that this is primarily a mod *manager* - making it easy to for new players to use mods is the top priority, and merging mods is pleasant side effect if and only if it can be done without compromising that goal.  Which means that a basic system comes first, then extending that to a smart parser. 

All that said, it would probably be useful to see ModBase if the code is open!

[Sean Mirrsen]

Yeah, I think I always kept it open. I really wasn't that good a coder then though (I used VISUAL BASIC for crying out loud), so the result isn't terribly great, but it has/had a lot of neat functions, not least of which is keeping track of your init file and reapplying your settings on every new DF release or install. Also switching tilesets (but not graphics sets). Little broken swiss army app, that.

Looking at the code now, it's also not terribly well commented, and yeah, will need a great big rewrite to be able to tell apart creature castes and keep them ordered. The base mod-manager functions should only need minor tuning though, as I doubt the process itself will be very different.

To clarify a bit: everything the OP says the standard should do, ModBase already does/did. It was intended, by me, to become a modding standard back then, since diffs are much easier to make, and its conditional content creation functions are incredibly useful sometimes. The only problem(s) with it is that it's currently broken in regards to creatures, and is written in relatively slow VB.NET by a relatively low-skilled programmer (me).

[thistleknot]

Well I'm a newb at qt but... If I had access and collaborators maybe we can port it over
.so far the app I'm working w has two panes side by side for file loading
 I figured I was gone a make it look more and more like mod manager but modbase sounds like a good candidate as well. I figured at some poi t I was gonna port some code over from another tool.

I don't know if anyone cares to learn qt w me cough cough "Sean". Especially if one knows VB already. Practically the same.

[Pidgeot]

I fully agree with the basic process, but I'm not sure about keeping the mods as full raw folders - at least not long-term. It is definitely the simplest approach to distribution, given the current state of the mod community, but it does bring about a few issues.

Distributing as a complete raw folder means that it's tied down to that single version of DF. If you have a mod for, let's say, 0.40.03, then the files need to be updated for 0.40.04 - even though the actual changes made by the mod may be the same. (if all the raws changed by the mod are identical between the two DF versions, you can still reuse it, of course, but assume that things change)

Suppose that we have a LNP-style folder structure for this: DF is stored in X\DF and the mods are stored in a folder X\LNP\Mods, and that our current installation has a specific mod M bundled (so X\LNP\Mods\M). If a new bundle is released for a new DF version, but without mod M (because it wasn't updated), then you have a problem if users still try to apply it - because the raws get reverted to the previous version. We cannot, IMO, expect users to always extract new bundles to a freshly created folder; many people will end up using the same base folder.

Distributing as a patch avoids this issue; if the mod changes can still be applied, then there's a good chance they still have the desired effect; if not, then the mod is probably broken anyway.

Nothing prevents us from starting with raw distribution, of course, and it's probably the fastest way for us to get started, but it's worth keeping in mind, and considering whether or not we really save much in the long run.

This does lead me to the next issue, which is perhaps the most important one - what does a patch actually look like? We basically have two options; a standard diff similar to the ones you get from existing diff utilities, or a custom format tailored to the raw file structure. A standard diff would require minimal effort, but it is potentially brittle when you add more mods, and it also forces mods to be very independent.

To see why this is a possible problem, assume we have a mod to remove aquifers, and a mod that adds material layers. If any of the added materials are listed as aquifers, then a standard patch format will have no way of removing that when the aquifers mod is added - and the aquifer tags that do get removed may well depend on the order those extra materials are added in, because the patch is unlikely to provide enough context.

The alternative is to create our own custom format - and that means deciding what we want from that format. The way I see it, we would need a proper model of the raw file format, and essentially treat patches as a script, which might look something like this:

Code: [Select]

[FILE:raw/objects/creature_standard.txt] (opens the specified filename)
[UNDER_TAG:CREATURE:DWARF] (finds [CREATURE:DWARF])
[CHANGE:NAME:3:dwarf:dwarves:dwarven:dorf:dorfs:dorfen] (changes the subtag [NAME:dwarf:dwarves:dwarven] to [NAME:dorf:dorfs:dorfen] - the 3 is to specify the first three values are for searching)

[FILE:raw/objects/inorganic_stone_layer.txt]
[REMOVE_ALL:AQUIFER] (removes all AQUIFER tags)

This is of course just an example off the top of my head; an actual format would likely be different.

A custom format is of course more complex, and requires more effort to get started, but I suspect it will be better in the long run - we can easily start with a standard patch and switch over later, but once we do want a custom format for the added flexibility, it is crucial that we do our best to get the format "right" (preferably making it extensible so we can handle use cases that haven't been considered earlier). We could still create a patch automatically; only mods that want to affect other mods would require the advanced stuff (and would need to do so manually).

Perhaps the quick approach is good enough for now, but we should at least be sure that the limitations are acceptable (until a better solution is made).

[Sean Mirrsen]

<snip>

ModBase uses a custom markup format that allows you to only distribute changes to raws, and you're only tied down to original raws in the sense that you might not get results you expect if the raws change too much between versions (i.e. trees in the recent version, creatures when castes and tissues were introduced, etc.)

The format allowed conditional creation of procedural entries, like creatures or weapons.
I.e. this:

[CREATURE:#_MAN]
[COND:BODY@1&QUADRUPED][COND:MEGABEAST!][COND:SEMIMEGABEAST!][COND:POWER!][COND:NAME]
[NAME:#man:@1#men:@1#man]
[CAN_SPEAK][CAN_LEARN][CANOPENDOORS]
[SIZE:#+1]
[BODY:QUADRUPED!!HUMANOID:>>]
[MUNDANE:!]
[PET:!]
[EQUIPS]
[COMMON_DOMESTIC:!]
[NATURAL]
[PREFSTRING:mystery]
[CHILDNAME:NAME@1#man child:NAME@1#man children]
[ATTACK:MAIN:BYTYPE:GRASP:punch:punches:1:2:BLUDGEON][ATTACKFLAG_WITH]
[ATTACK:MAIN:BYTYPE:STANCE:kick:kicks:1:3:BLUDGEON][ATTACKFLAG_WITH]

This mod combs through all creature entries loaded at the point where it's applied, and looks for several conditions (it must have QUADRUPED in its body tag, it must have a NAME, it must not be a MEGABEAST, SEMIMEGABEAST, or POWER). For each creature that matches these conditions, it creates a new creature entry called <CREATURE>_MAN, switches its QUADRUPED body for a HUMANOID, strips it of MUNDANE, PET, and COMMON_DOMESTIC tags, increases its size by 1, gives it ability to learn, speak, open doors and equip equipment, and gives it new names and attacks to match. The mod was, appropriately, named after Dr.Moreau.

Well, it used to do that until all the tissue and caste changes, anyway. -_-

The markup generally works, and mods that only add stuff can be plugged into it as-is, since regular tags and entries are just added in.

[PeridexisErrant]

I fully agree with the basic process, but I'm not sure about keeping the mods as full raw folders - at least not long-term. It is definitely the simplest approach to distribution, given the current state of the mod community, but it does bring about a few issues.

Distributing as a complete raw folder means that it's tied down to that single version of DF. If you have a mod for, let's say, 0.40.03, then the files need to be updated for 0.40.04 - even though the actual changes made by the mod may be the same. (if all the raws changed by the mod are identical between the two DF versions, you can still reuse it, of course, but assume that things change)
<snip>
Perhaps the quick approach is good enough for now, but we should at least be sure that the limitations are acceptable (until a better solution is made).

Key point:  good enough for now, and basic process.  Until we have a working basic example this is mostly academic. Once we have a manifest file, it's easy enough to note which version the mod is based against.  If the pack then also includes old vanilla mods, either entirely or as a backwards patch, you can then diff against that and it should work unless big changes are made.  In any case major-raw-changing updates are very rare, and thinking too much about them just because we're currently in that period would be a mistake. 

Again though, the problem is not advanced merging tools or special syntax. 

The problem is that no new players use them. 

Instead, I'm proposing a simple format which *already almost exists*.  The only difference is that many files in a pre-installed mod can be automatically deleted by the utility; content already exists and is compatible.  The key point for new players is an easily loaded collection, not the advanced merging. 

[expwnent]

In the most recent version of DFHack, the raw/scripts folder will be checked for Lua scripts before the hack/scripts folder. This means that each savegame can have its own collection of scripts that come with it and you can transfer saves more easily between people that don't have the same mods. It also means that for this project you can just throw all the scripts into the same folder as long as there's no overlap and it'll work fine.