Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Meph]

What happens to mutually exclusive mods?

One removes kimberlite, the other adds new uses for diamonds (glass cutting for example). No kimberlite = No diamonds. While the Raws would fit and the program would merge them, you would have an unuseable mod in the end.

[thistleknot]

Think about this simple case: 1 mod adds a creature specific token at the end of a creature definition. another adds a creature. If you add them in the wrong order, you could get the creature specific token to the wrong creature if you don't order the merge right.

I think some simple way to either derive a precombined patch

or to keep track of line changes as you add/remove lines would b in order to address this concern

Basically, you would check against vanilla, and determine if the changes are merely additive or subtractive.

Then if they are additive, it's all green light.  Merge with rest of patches (barring any duplicate line additions?)

Yeah, it get's complicated apparently...

I was just assuming [by] always checking a patch against vanilla [we] would [edit:] be basing [the] "Additive" part of the mod [against Vanilla]; and if you could somehow map the BEFORE and AFTER line states of working mod vs vanilla, you could somehow accommodate future line injections based on tracking insertion points or something.

but apparently, it gets complicated, and one might as well do some object tracking at that point.

[King Mir]

Think about this simple case: 1 mod adds a creature specific token at the end of a creature definition. another adds a creature. If you add them in the wrong order, you could get the creature specific token to the wrong creature if you don't order the merge right.

I think some simple way to either derive a precombined patch

or to keep track of line changes as you add/remove lines would b in order to address this concern

Basically, you would check against vanilla, and determine if the changes are merely additive or subtractive.

Then if they are additive, it's all green light.  Merge with rest of patches (barring any duplicate line additions?)

Yeah, it get's complicated apparently...

I was just assuming always checking a patch against vanilla would the base "Additive" part of the mod.  And if you could somehow map the BEFORE and AFTER line states of working mod vs vanilla, you could somehow accommodate future line injections based on tracking insertion points or something.

but apparently, it gets complicated, and one might as well do some object tracking at that point.

One mod adds [PET] at the end of a vanilla creature. Another mod adds a new creature at the same spot. Both are merely additive. But if merged in the wrong way, the result would add [PET] to the new creature instead of the vanilla creature.

[King Mir]

What happens to mutually exclusive mods?

One removes kimberlite, the other adds new uses for diamonds (glass cutting for example). No kimberlite = No diamonds. While the Raws would fit and the program would merge them, you would have an unuseable mod in the end.

That's a rather benign error. The result is still a valid set of raws. So it's not a big problem.

Also, since the primary use of this tool is for including a lot of minor mods, hopefully 2 minor mods that only make those changes would obviously conflict from their description, so nobody would actually try to do that.

[King Mir]

Here's the code to merge mod files in Python. It merges non-conflicting sequences as proven by the previously published can_merge_seq function.

Neither of the functions can handle two mods adding the same content yet. Also, this still needs to be hooked up to PeridexisErrant's code.

Code: [Select]

import difflib

def do_merge_seq (mod_text, vanilla_text, gen_text):
    if vanilla_text == gen_text: #this should happen often
        return mod_text
    van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)   
    van_gen_match = difflib.SequenceMatcher(None, vanilla_text, gen_text)   
   
    van_mod_ops = van_mod_match.get_opcodes()
    van_gen_ops = van_gen_match.get_opcodes()
   
    output_file_temp = []   
    cur_v = 0
    while cur_v < len(vanilla_text) :
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        #print van_mod_ops[0]
        #print van_gen_ops[0]
        #print cur_v
        if mod_tag == 'equal' and gen_tag == 'equal' :
            if mod_i2 < gen_i2:
                output_file_temp += vanilla_text[cur_v:mod_i2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
            else:
                output_file_temp += vanilla_text[cur_v:gen_i2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)           
        else:                                                 
            if mod_tag != 'equal' :
                output_file_temp += mod_text[mod_j1:mod_j2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_gen_ops.pop(0)   
            elif gen_tag!='equal':
                output_file_temp += gen_text[gen_j1:gen_j2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)   
            #if neither gen_tag nor mod_tag is 'equal', this mod can't be merged
        #print (output_file_temp)
    if van_mod_ops:
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        output_file_temp += mod_text[mod_j1:mod_j2]
    if van_gen_ops:
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        output_file_temp += gen_text[gen_j1:gen_j2]
    return output_file_temp

vanilla_file = "vanilla"
output_file = "vanilla"

print ("----")
print output_file
output_file = do_merge_seq ('anything at all', vanilla_file, output_file)
print output_file
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('nilla', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vanill', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('vonilla', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vonilla', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vanilla', vanilla_file, output_file)
print ''.join(output_file)


[hermes]

I don't understand why you guys are beating around the bush with this.  The way PE wants the version 1 to work you should be able to detect all these conflicts but you have to refuse the merge each time.  There is no way a blind diff patch can merge any mod that takes something away, guaranteed error free.

Appealing to PE again... you're in a much better position than I ever have been to impose a standard for mods, because the starter pack you curate is so popular.  I don't see what the rush is, if you spend some time trying to set up a single, decent system that is flexible enough to handle the diverse range of DF mods, that would be great.

However, I do agree that there should be a non-manifest way of merging, so I'd go the route of making a two tier mod-integration system where mods without a manifest are given lower priority and are more readily excluded if they remove objects from the raws.  As everyone seems to be slowly realizing, there are so many potential conflicts you have to go all the way to actually make the thing work properly.

That said I feel like I'm trolling a bit, so I'll bow out here for now, I guess you guys are going in a different direction than I'd anticipated.  Really hope it works out, good luck!   

[PeridexisErrant]

https://github.com/PeridexisErrant/Py-Mod-Loader

Now on GitHub, for easier collaboration and code compilation.  Note that the readme etc is not at all finished, but eventually I hope that'll have the definitive standard descriptions.

I don't understand why you guys are beating around the bush with this.  The way PE wants the version 1 to work you should be able to detect all these conflicts but you have to refuse the merge each time.  There is no way a blind diff patch can merge any mod that takes something away, guaranteed error free.

Appealing to PE again... you're in a much better position than I ever have been to impose a standard for mods, because the starter pack you curate is so popular.  I don't see what the rush is, if you spend some time trying to set up a single, decent system that is flexible enough to handle the diverse range of DF mods, that would be great.

However, I do agree that there should be a non-manifest way of merging, so I'd go the route of making a two tier mod-integration system where mods without a manifest are given lower priority and are more readily excluded if they remove objects from the raws.  As everyone seems to be slowly realizing, there are so many potential conflicts you have to go all the way to actually make the thing work properly.

That said I feel like I'm trolling a bit, so I'll bow out here for now, I guess you guys are going in a different direction than I'd anticipated.  Really hope it works out, good luck!   

It's a problem all right.  The solution seems to be either living with some working merges that produce non-functional raws, disallowing removals, only merging one mod, or full raw comprehension.  I think occasional errors is probably the way to go, and just warn people to be careful with mods.  I like the idea of trusting mods with a (non-tool-written) manifest more, we could do good stuff with that. 

There's no way I'm including a non-final system in the starter pack; that way lies madness through support enquiries.  You're probably right about imposing a standard - I don't want to end up making the IE6 of mods though, so there's a lot of care required.  No rush at all. 

I see no trolling, and it's good to hear an alternative view - especially from someone who can see us falling into traps.  Please stick around! 

[thistleknot]

Well, the beauty of a mod system is this.

If you keep it to simple things like base ascii game changes, and offer as many mods as you can.  You keep the mods down to their ascii elements and allow players to choose from a wide range of mods that are applied on a batch level.

I mean, players can simply just check "one" mod and play that without ever wanting to merge them.

Barring most likely dfhack, unless of course you have dfhack and the flattening of raw files doesn't break dfhack.

and maybe last you offer a tileset change on top.

The "mix" of mods can be left up to the community.  Supposedly the system can work as long as one derives a diff from base to their mod, then the mod can be applied using this patch system.

So in the end, regardless of whether the "merging of mods" works right or not, your allowing modders to port/be included in a mod system by merely supplying patches, what's great about that, is it's possible to extract patches ourselves and submit them using a git system.

If players so choose, the "merge" process can come with a disclaimer and "no silver bullet" kind of situation.

[Button]

I'm still completely in agreement with Hermes by the way, I just think that it's a conclusion y'all won't come around to until you've grappled with the problem a bit more . Until then, I'll help the diff-based approach as best I can.

I was thinking I might write a last-pass errorchecker which could do some basic raw comprehension and catch errors which the diff might miss - finding dupes across files, reactions with no possible reagents, things like that. Could be expanded into a more fully-functional syntactic parser later. It'll give me an excuse to pick up Python again - I've been stuck doing Java at work.

[PeridexisErrant]

I was thinking I might write a last-pass errorchecker which could do some basic raw comprehension and catch errors which the diff might miss - finding dupes across files, reactions with no possible reagents, things like that. Could be expanded into a more fully-functional syntactic parser later. It'll give me an excuse to pick up Python again - I've been stuck doing Java at work.

You know, I was thinking that a post-processing pass might be the best way to catch some of this stuff, but I'm nowhere near making it happen.  That would be awesome!

I make no comment on the accuracy of observations that we may have underestimated the problem just a little 

[Merkator]

This short script should extract every single token in file. Maybe someone find it useful.

Code: [Select]

import re
import sys

if len(sys.argv) < 2:
    filepath = str(input("FILEPATH> "))
else:
    filepath = sys.argv[1]


def reader(filepath):
    tokens = []
    name = ""
    with open("test.txt", 'r') as file:
        for line in file:
            if name == "":
                name = line
            for i in re.findall("\[[A-Za-z0-9:]*?\]", line):
                tokens.append(i)
    return (name, tokens)


def main():
    name, tokens = reader(filepath)
    for i in tokens:
        print(i)

if __name__ == "__main__":
    main()


BTW why not use something like robocopy \mir or cp -rvu for major mods.
Or just add small manifest file with information about compatibility. Number of major mods is not that big after all.
They are maybe 10 really big mods and no one will try to merge Masterwork with MLP 

Code: [Select]

  {
     "name":<name of the mod>,
     "version":<version of the mod>,
     "compatible-major":{
         "name-of-major-mod":"version-of-major-mod"
  }
}

Something like this. Where * is wildcard.

{
  "name":"Mycrazymod",
  "version":"0.0.1",
  "compatible-major": {
     "MasterworkDF":"5.*"
   }
}


For smaller mods you insist on merging raws itself. Why not merge diffs first?

[thistleknot]

Update5
Asking if it's possible to apply nway merging on a patch level
http://stackoverflow.com/questions/25408998/octopus-merge-using-patch-files

Update4
https://github.com/thistleknot/df_40_09_Flattened/commits/3-wayResults of merge of advciv, plantfixes, and direforge (ascii) & flattened here:

I can do 3 way merges of conflicting mods if they share a common ancestor.

Code: [Select]

C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]> git branch 40_09_DireForge-Flattened f18f255
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]> git merge 3459e01
Auto-merging raw/objects/plant_standard.txt
CONFLICT (content): Merge conflict in raw/objects/plant_standard.txt
Auto-merging raw/objects/entity_default.txt
CONFLICT (content): Merge conflict in raw/objects/entity_default.txt
Automatic merge failed; fix conflicts and then commit the result.
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way +48 ~23 -0 !2 | +0 ~0 -0 !2]>
It's a git branching issue.  I DO NOT KNOW if it can be done offline.  What this really is, is a way for me to merge a ton of mods now.

Update3
Figured out how to do an octopus Merge, it was about ensuring I had a common ancestor to base it on (didn't realize anything about a common ancestor until I looked at the rejected merge* files that served as the 3 way comparison), and the common ancestor HAD TO BE flattened vanilla.  Which was a bit of work.  because the mods are applied, then flattened.  So my prior common ancestor was non flattened ascii vanilla...  But I rebranched based on flattened ascii, replaced with already flattened advciv and plantfixes, and I was able to merge two mods together at the same time

Code: [Select]

C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way +2 ~0 -0 !]> git merge 514bde0 13a563f
Trying simple merge with 514bde0
Trying simple merge with 13a563f
Merge made by the 'octopus' strategy.
 raw/objects/creature_darkdwarf.txt        | 363 +++++++++++++
 raw/objects/creature_firedwarf.txt        | 363 +++++++++++++
 raw/objects/creature_frostdwarf.txt       | 362 +++++++++++++
 raw/objects/creature_stonedwarf.txt       | 362 +++++++++++++
 raw/objects/creature_stormdwarf.txt       | 362 +++++++++++++
 raw/objects/creature_wilddwarf.txt        | 362 +++++++++++++
 raw/objects/descriptor_shape_standard.txt |  25 +
 raw/objects/entity_darkdwarf.txt          | 821 +++++++++++++++++++++++++++++
 raw/objects/entity_default.txt            |   9 +
 raw/objects/entity_firedwarf.txt          | 815 +++++++++++++++++++++++++++++
 raw/objects/entity_frostdwarf.txt         | 817 +++++++++++++++++++++++++++++
 raw/objects/entity_stonedwarf.txt         | 822 ++++++++++++++++++++++++++++++
 raw/objects/entity_stormdwarf.txt         | 815 +++++++++++++++++++++++++++++
 raw/objects/entity_wilddwarf.txt          | 816 +++++++++++++++++++++++++++++
 raw/objects/plant_crops.txt               | 218 +++++---
 raw/objects/plant_garden.txt              | 169 +++++-
 raw/objects/plant_standard.txt            | 128 ++++-
 raw/objects/reaction_plantfix.txt         |  92 ++++
 18 files changed, 7618 insertions(+), 103 deletions(-)
 create mode 100644 raw/objects/creature_darkdwarf.txt
 create mode 100644 raw/objects/creature_firedwarf.txt
 create mode 100644 raw/objects/creature_frostdwarf.txt
 create mode 100644 raw/objects/creature_stonedwarf.txt
 create mode 100644 raw/objects/creature_stormdwarf.txt
 create mode 100644 raw/objects/creature_wilddwarf.txt
 create mode 100644 raw/objects/entity_darkdwarf.txt
 create mode 100644 raw/objects/entity_firedwarf.txt
 create mode 100644 raw/objects/entity_frostdwarf.txt
 create mode 100644 raw/objects/entity_stonedwarf.txt
 create mode 100644 raw/objects/entity_stormdwarf.txt
 create mode 100644 raw/objects/entity_wilddwarf.txt
 create mode 100644 raw/objects/reaction_plantfix.txt
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]>
Update2
http://stackoverflow.com/questions/5292184/merging-multiple-branches-with-git
Apparently the first octopus merge didn't work very well, but a more indepth explanation is here:

Code: [Select]


    git checkout master
    git pull origin feature1 feature2
    git checkout develop
    git pull . master (or maybe git rebase ./master)



The first command changes your current branch to master.

The second command pulls in changes from the remote feature1 and feature2 branches. This is an "octopus" merge because it merges more than 2 branches. You could also do two normal merges if you prefer.

The third command switches you back to your develop branch.

The fourth command pulls the changes from local master to develop.

Hope that helps.

EDIT: Note that git pull will automatically do a fetch so you don't need to do it manually. It's pretty much equivalent to git fetch followed by git merge.

Update:
http://stackoverflow.com/questions/25388806/github-3-way-merge-patch

Someone may have given me an answer which echo's another person's answer on the same issue, Octopus Merge:

Code: [Select]

git merge B C

This short script should extract every single token in file. Maybe someone find it useful.

We already hit on the token extraction two ver... Both sed regex commands; , one that derives tokens using regex like ur ex then injects filename on top, and another that puts tokens on their own lines effectively keeping comments. Both ver remove all whitespace.

Please do tell. How does one combine diff's? I would love to know

Combeindiff combines two sequential commits of the same branch, however these patches are all individual branches off the same base, vanilla. So far the best solution I've had is interdiff and I have not figured out how to apply an interdiff merged patch file.

[Meph]

Silly question among all this talk of different technical problems and solutions: Has anyone asked around, written some PMs to mod authors about writing and packaging their mods in a standardized system?

Because I havent seen a single modder here, besides Putnam and me, and both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.

[Button]

Silly question among all this talk of different technical problems and solutions: Has anyone asked around, written some PMs to mod authors about writing and packaging their mods in a standardized system?

Because I havent seen a single modder here, besides Putnam and me, and both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.

What am I, chopped liver?

[Meph]

A button?

No, sorry, I dont connect your name with any mods I recall... and there are no threads in the mod release board by your name. So as unfortunate as it is, I have to declare you a button.