Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Pidgeot]

The key point for new players is an easily loaded collection, not the advanced merging.

I agree, but that's exactly why the standard patch format might not be good enough.

Yes, distributing mods as full raw folders and using standard diff utilities is easy, and we could get something running without too much effort - but it comes with a cost: the patches don't know anything about the raw structure, and that makes it that much more difficult to make sure everything works as intended when you deal with multiple patches.

A standard patch basically looks for a specific part of the original file. This is generally annotated with a line number to provide a reference point, but that line number can be flexible to accomodate other changes.

What this means is that if mod A adds 10 creatures to the middle of a file, then the patch for mod B, which changes something about other creatures, might end up modifying one of those newly added creatures - even though it wasn't supposed to. We have no way of knowing when this happens - we only know whether or not the patch was applied to something.

Now, this may indeed be acceptable for the first implementation - I don't use mods, so I don't know if this is a common case or not. If it's good enough, then it's good enough, and I have no objections beyond that - I just think it's important to at least be aware of the possible problems.

[King Mir]

I've been thinking I need such a too for a while, and recently considering writing my own. Everything In the OP makes sense to me. The biggest problem is dealing with conflicts. The OP says you should abort on conflict, but that requires being able to detect the conflict first. Also, I think it's really limiting if two mods cannot modify the same vanilla file. When adding content, this is relatively easy, but with mods that remove content, you can run into problems, and it can be hard to detect them.

One thing that could help is if there are different settings for applying a mod. So one setting would apply a diff to each file that it modifies that already exists. Another would instead overwrite any conflicting files; this is useful for situations like sorting gems, because the order gems are loaded affects their appearance in the jewelers workshop. And you might need a third setting for some other special cases.

[Sean Mirrsen]

So I'm guessing that all of this means that I'm going to have to resurrect ModBase after all. Everyone seems to be dancing around in circles over the functionality that ModBase already mostly provided back when it worked. All it really couldn't do on its own was extract the mods by itself. *sigh* I do not really have the time for it nowadays.

Heh, I even had a name for that project. Clean Slate. Might start a repo up on GitHub and see if I can make any progress...

[King Mir]

@Dirst:  Yep, sounds like we're on the same page. 

How's this for a set of goals?
v0.1 - basic logic in place, folders findable, etc.  Written in Python for PyLNP compatibility
       - can derive a diff correctly and create a set of raws with one mod-patch applied
v0.2 - installs multiple non-conflicting mods correctly
       - handles errors and conflicts gracefully
       - write log file with mod merge order etc
       - place holder handling for a manifest file (note existence)
v0.9 - GUI time!  Integrated as a tab in the PyLNP - get it functional in new context, refactor to fit, etc.
    .1 - implement manifest and use information from it in display (if present)
       - 'simplify mod folders' option ala LNP, deleting extra files (not readme etc, but eg rest of DF install)
v1.0 - start finding or soliciting or formatting some mods

I think the steps would be:
v0.1 - able to apply and remove any one mod.
v0.2 - able to merge any number of mods as long as no two mods modify the same file
       - detect duplicate raws.
v0.3 - allow scripts: a mod that is actually a shell script that is applied naively
v0.4 - allow mods to modify the same file, start to manage conflicts
... - more as needed

The GUI can be worked on in parallel.

[PeridexisErrant]

I like the principle of NO SPECIAL CASES.  No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws.  If that makes some advanced stuff impossible, simple is better.  (I've been reading this list)

As to detecting merge conflicts, there are two ways to do it - both of which work to combine files.  The fast and simple way is the standard merge conflict test; if your diff returns an error show the problem in the live preview. 

The slow way, which will return no false negatives, is to add and subtract combinations of diffs to check that changes don't overwrite.  For example, we take vanilla DF and apply mods by diffs A, B, and C in that order.  We can then check for problems by confirming that VABC-A == VBC, VABC-B == VAC, skip the case VABC-C == VAB as trivially true, and also check VABC-AB == VC for completeness.  Given that this is N-1 factorial checks for N mods, it could be too slow for a live preview but if we only call it when the fast merge check returns OK that should be acceptable.  If one passes and the other fails, I'd probably alert the user and allow them to decide. 

Modbase, rubble, Thistleknot's project are all awesome but that's not what new players need.  They don't need amazing merge tools.  They just need an easy way to try some mods, and the LNP provides the model I want for graphics and utilities already.  If you can only use one mod at a time that's not a fatal flaw for these users! 


@King Mir:  good points;
    0.2 - Files that are identical to vanilla should be detected and removed, though this might come later (not 0.9 though, you're right)
         - Any test versions that overwrite files instead of using diffs would be pre-0.1
    0.3 - No.  No special formats in the mod.  Not until 2.0 minimum, and even then it has to be backwards compatible
    0.4 - As above, I foresee using diffs early.  Could change if it's harder than I expect, but unlikely. 

[Sean Mirrsen]

If they want to try out one mod, then distributing mod diffs will work, as well as distributing whole raw folders. You don't need a program to install one mod, unless it's a mod with lots of stuff that you might not want all of.

ModBase started as a commandline script that installed or removed my Martial Arts and Minerals mods with various optional elements for people that didn't want certain things I added. It just evolved into a mod manager from there, and soon became the only real way to install the mods I made because I developed them using the program's functions.

[King Mir]

I like the principle of NO SPECIAL CASES.  No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws.  If that makes some advanced stuff impossible, simple is better.  (I've been reading this list)

As to detecting merge conflicts, there are two ways to do it - both of which work to combine files.  The fast and simple way is the standard merge conflict test; if your diff returns an error show the problem in the live preview. 

The slow way, which will return no false negatives, is to add and subtract combinations of diffs to check that changes don't overwrite.  For example, we take vanilla DF and apply mods by diffs A, B, and C in that order.  We can then check for problems by confirming that VABC-A == VBC, VABC-B == VAC, skip the case VABC-C == VAB as trivially true, and also check VABC-AB == VC for completeness.  Given that this is N-1 factorial checks for N mods, it could be too slow for a live preview but if we only call it when the fast merge check returns OK that should be acceptable.  If one passes and the other fails, I'd probably alert the user and allow them to decide. 

Modbase, rubble, Thistleknot's project are all awesome but that's not what new players need.  They don't need amazing merge tools.  They just need an easy way to try some mods, and the LNP provides the model I want for graphics and utilities already.  If you can only use one mod at a time that's not a fatal flaw for these users! 

Are you saying to use diff to see if two mods modify the same file, or if diff3 returns an overlap? Seeing if two mods modify the same file doesn't need diff: you can assume they do if both include the same file. Testing if diff3 returns no overlap would need to be done for every combination of added mods, so you still have N factoral checks. But you're just dealing with a small number of text files, so maybe performance won't be an issue.

You also need to support the options LNP already supports. That means being able to apply a remove aquifer patch and a remove exotic animal patch on top of a graphics pack. Then you want to have a mod add a creature or stone type to that. That's a baseline for which conflicts this tool would need to be able to manage. I concede that you want to keep things simple, but can you?

I do think it's important that the only thing a Mod needs to be included is the files that would override the standard raws, and that you don't invent a DSL for writing a mod. 

[thistleknot]

One way I was going g to deal w mod conflicts was...

To do the merge conflicts manually, derive a patch between vanilla and my now manually merged mod.

Only issue is. This has to be done ahead of time and it would mean a modder would have to manually do the merge.  However... It is possible but its no different than doing it manually it's just you have a patch file afterwards. Case in point. Accelerated mod + modest mod changes both alter creature and things like arcvision and clutch size at the same spot in creature files... So... A modder could make that patch file afterwards, but this isn't an ideal solution.

[PeridexisErrant]

@King Mir
I'm talking about overlapping diffs, not simply diffs in the same file; the latter is as you note unproblematic. Never underestimate the impact of a factorial performance hit - the problem is that it doesn't take many more than the test case to cause a big problem.  It's (n-1)! too, because the last diff on can always be reversed.

Other things like the aquifer tags should be fine, as they're not done by merging raws but rather by editing them based on tags. Which *will* remain a separate function.

@Thistleknot - modders producing manually combined mods (perhaps with the merger tool) is not a particularly elegant solution, true. However it's no worse than the current situation and I think substantially better than making the format and workings of this system any more complicated than it has to be.

Graphics are complicated enough that I'm just ignoring the issue for now. I'll think of something clever for that later. Many ideas but nothing worth sharing yet.

[Pidgeot]

For example, we take vanilla DF and apply mods by diffs A, B, and C in that order.  We can then check for problems by confirming that VABC-A == VBC, VABC-B == VAC, skip the case VABC-C == VAB as trivially true, and also check VABC-AB == VC for completeness.  Given that this is N-1 factorial checks for N mods

How did you arrive at (n-1)!? By my calculations, you create the expected final set of raws (N patches), N sets of raws where each set skips 1 patch (N-1 patches per set), and N sets of raws where you "unpatch" one patch from the final set (N "unpatches"). That's 2N+1 sets of raws, N^2 patches applied, and N patches unapplied.

[Dirst]

I like the principle of NO SPECIAL CASES.  No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws.  If that makes some advanced stuff impossible, simple is better.  (I've been reading this list)

I think we can go one or two steps past vanilla diffs without getting bogged down.

First, location hints need to be tags rather than lines.  This handles all file collisions unless they try to modify the same tag.  It might help if the vanilla raws were pre-flattened to have one tag per line.

Second, allow for a position hint to be a regular expression.  In other words, the scripting version of a mod becomes a slightly hardened sed engine.  This is extraordinarily powerful (for example, one can write a regular expression that matches a syndrome and any possible subtags under it) and puts the onus entirely on the modder rather than the player. 

A location hint of \[CREATURE:ELEPHANT\].*?\[PETVALUE:[0-9]+?\] will find the petvalue of an elephant, no matter what it has been changed to by another mod.  A sed-like like of s/(\[CREATURE:ELEPHANT\].*?\[PETVALUE:)[0-9]+?\]/\1666\] changes the petvalue of an elephant to 666 (in tribute to the elephants in 23a).  The hardening is only necessary if you want to do something other than silently skip any failed matches.

With these two features in place, it should be possible to handle graphics packs like any other mods.  The one thing it does not do gracefully is re-order things within a file, such as sorting the gemstones.

[Sean Mirrsen]

I did all that... with simple strings... I could reference certain values of certain tags of certain objects, like creating a sword with a damage value dependent on that of an existing sword.

[ITEM_WEAPON:ITEM_WEAPON_SSWORD_CRUEL]
[NAME:shortsword:shortswords]
[ADJECTIVE:Cruel]
[DAMAGE@:%ITEM_WEAPON_SWORD_SHORT+10:GORE]
[WEIGHT:35]
[SKILL:SWORD]
[CRIT_BOOST:1]
[TWO_HANDED:4]
[MINIMUM_SIZE:4]
[MATERIAL_SIZE:6]
[STICK_CHANCE:30]

It's context-sensitive in this case (pulling the damage value from the same spot of the same tag of a different item of the same type), but was intended to work just as well for different-type objects and different tags.

[Pidgeot]

I like the principle of NO SPECIAL CASES.  No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws.  If that makes some advanced stuff impossible, simple is better.  (I've been reading this list)

I think we can go one or two steps past vanilla diffs without getting bogged down.

The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.

[King Mir]

@King Mir
I'm talking about overlapping diffs, not simply diffs in the same file; the latter is as you note unproblematic. Never underestimate the impact of a factorial performance hit - the problem is that it doesn't take many more than the test case to cause a big problem.  It's (n-1)! too, because the last diff on can always be reversed.

Other things like the aquifer tags should be fine, as they're not done by merging raws but rather by editing them based on tags. Which *will* remain a separate function.

So effectively you're allowing modding scripts as long as they are run after your tool, with aquifers being an example of such a script.

If you assume that if A and B don't conflict with C, then A+B don't conflict with C, then you reduce your checking to N^2. N is also quite small. So you might be ok.

You have a very strict requirement for non-conflicting mods. The approach I had in mind would instead have mods have a specified order of loading, so that a later mod overrides a prior mod.

And BTW- I think the suggestion to have one tag per line is a good one. You can preprocess the mod and vanilla raws, so that each tag is on one line and strip comments before feeding it to your differ.

[Dirst]

I like the principle of NO SPECIAL CASES.  No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws.  If that makes some advanced stuff impossible, simple is better.  (I've been reading this list)

I think we can go one or two steps past vanilla diffs without getting bogged down.

The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.

I think some kind of custom format is unavoidable; it just needs to be open.

We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install.  We also don't get much bang for the buck if we don't store things as diffs of some type (a graphics pack can make tiny changes to lots of large files).