Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Button]

A button?

No, sorry, I dont connect your name with any mods I recall... and there are no threads in the mod release board by your name. So as unfortunate as it is, I have to declare you a button.

Just cause I don't plaster my work all over my signature...

Plant Bugfix/Minor Mod is mine. You'll notice it has a thread on the mod release board.

[Merkator]

If I am wrong somewhere in the the middle please correct me.
The biggest diff issue is that, when you apply one mod as diff file to the vanilla, every single mod or at least most of them invalidates.
That's because line numbers change.
3-way diff sound pretty nice, but still the problem with conflicting mods stay.

Of course the best thing is to have just context dependent diff format, and not format that depend on line numbering.
Or format that use relative line numbering. For example apply some patch after n lines from point x.

And I think using git is not the best option here. Git is great, but IDK if this is best use case for it, but if you insist. ;P


Why not just squash all patches in to single diff like that http://stackoverflow.com/questions/616556/how-do-you-squash-commits-into-one-patch-with-git-format-patch?rq=1
 

[thistleknot]

Squash across branches?

squashing is for commits of the same branch.

we are aware that the lines and what not change with each successive patch btw.  The only way to address that is to do some type of octopus merge...  That's why I was hitting on this octopus merge solution.  If there was a way to ship git and derive patches from a base + a bunch of mods... that would be neat.  but I think it's a little much...

[thistleknot]

Octopus Merges vs Sequential Patching

Contextual differences between mods will break when:

After flattening/parsing:

creatureA and creatureB look the same around say 10-20 lines in their token make-ups.

So a mod comes in and INSERTS a new creature before creature's A and B.

So now the line references are messed up, diff will most likely fall back on a contextual match at this point for any creature mods that affect creature a or b in the lines I mentioned above.

So a mod, say modCreatureB, comes in, (which is a diff from vanilla:modCreatureB), mods creatureB at some odd lines.

Now, [the patch program when patching,] modCreatureB is confused since line #'s have changed, so it does a contextual match.

This is where the issue arises: If creatureA and creatureB had token makeup's similar in the same area, the contextual match will most likely fall on the first match or something.

Solution
Keep track of lines changes (Which I believe King Mir said he was doing)

OR...
Better solution IMO (which requires some working version of offline git command line batch scripting, we would ship a repository with the front end tool basically).

Octopus Merges
The mod applied will always be compared with the original base vs the last mod applied.  When conflicts occur with other mods... the injection point will always be derived from looking at the base, and seeing what other changes are being made (insertion/deletion points), and MATCHING those [insertion/deletion] points to the before/after contextual locations of the original patch file makeup.  The reason this works is the way CONTEXTUAL DIFFS ARE OUTPUTTED.  They always show BEFORE/AFTER CHANGES.  As long as we can MATCH 1 of those sides.  We can insert based on one of those side's matches + also being able to reference the line and be able to keep track of any changes based on the way the other mods are applied and line changes are moved around.

I would think...

It's entirely possible that other mods change multiple tokens within an object at different spots, therefore destroying contextual matchings...?  Idk

However, ultimately.  If one wanted to, object tracking is what would resolve that.

Anyways, my point being, is if we can just match ONESIDE of the contextual match while ensuring we're at the right line #, we should be able to do a clean insert or delete option.

And on a final note:
Alphabetizing
I've been pondering this idea on how to alphabetize raws to address the issue of TC mods or mods that completely reorder the contents of the files.

It may be possible with regular expressions (sed, or python, or grep), but unfortunately, the solution I'm thinking of involves breaking comments that are listed right before an object would be broken apart from the next object... so, I'd rather just remove comments than try to parse them using some batch script.

[Dirst]

Highly detailed explanation

This is why I mentioned the idea of bookmarking an intermediate file, which fixes all issues with line numbers.  It does not perform well if two mods want to insert different stuff at the same point (it'd likely throw a false-positive conflict) and it completely craps its pants if the file is re-ordered, but it looks like it should be relatively simple and get us a fair percentage of what we need.

We can include a simple special case that two mods adding at the same point are not considered a conflict (important for adding custom buildings and reactions to entity_default.txt), but I don't think PE is looking to make the pinnacle of file merger technology especially if manifest files are adopted.  A manifest can declare dependencies and known conflicts and "recommended" load orders relative to other specific mods.

[Dirst]

Alphabetizing
I've been pondering this idea on how to alphabetize raws to address the issue of TC mods or mods that completely reorder the contents of the files.

It may be possible with regular expressions (sed, or python, or grep), but unfortunately, the solution I'm thinking of involves breaking comments that are listed right before an object would be broken apart from the next object... so, I'd rather just remove comments than try to parse them using some batch script.

A multi-line match for regular expressions should be able to scoop up everything between tags and associate it with the next valid tag.  The special cases are (1) the header up to the [OBJECT:FOO] token, (2) header comments after [OBJECT:FOO] but before the first [FOO:BAR] and (3) comments at the end of the file.  (1) and (3) should be easy.  We just have to pick a standard for (2), is it associated with the header or the first object?

[thistleknot]

well, I demonstrated earlier how the file_name can be injected back to the top of the .txt file... so that isn't a concern

If you say 1 and 3 are easy.

I think the inbetween ones can be matched to the last token they were adjacent to...

but... that only works BEFORE whitespace is removed.

If whitespace is removed... then there won't be blank lines inbetween tokens...

And... it's hard to say how to split comments.

I mean..

[token]

comment

comment

[object:bar]

can be tricky as well
Ultimately though, comments aren't important [for playing] other than for exporting merges.

[thistleknot]

So someone got back to me on how a 3 way merge can be done using patch files, they said it was kind of hard to do without Subversion Control.   But said would be possible with diff3 and the source folders of each version to be changed (which can be achieved by extracting and applying patch files).

http://stackoverflow.com/questions/25408998/octopus-merge-using-patch-filesg

Code: [Select]

Otherwise, a 3-way merge is probably going to require the source directories of each version and then in a shell script:

mkdir VersionABC
for i in (cd VersionA && ls -R); do
  if [ -f VersionA/$i ]; then
    diff3 --merge VersionB/$i VersionA/$i VersionC/$i > VersionABC/$i
  else
    mkdir -p VersionABC/$i
  fi
done


I'm not 100% if they meant VersionB/$i VersionA/$i VersionC/$i vs ABC but it could be tested out.

Here's some documentation on diff3.

Looks like it does exactly what we have been concerned with, merge 2 descendents of a common ancestor.
http://www.chemie.fu-berlin.de/chemnet/use/info/diff/diff_8.html

Update
apparently diff3 suck at merging.

It had LARGE swathes of sections that were repeated, so nm on that idea.
Actually, using the -m option, such as diff3 -m a b c > a-bc.patch seems to work pretty well.

Great tutorial/explanation of how diff3 works

http://www.thegeekstuff.com/2012/08/diff3-examples/

[Putnam]

both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.

Pfah. My mod collection includes Sparking already. Including Masterwork stuff would simply be an ordeal rather than impossible at this stage.

[PeridexisErrant]

Silly question among all this talk of different technical problems and solutions: Has anyone asked around, written some PMs to mod authors about writing and packaging their mods in a standardized system?

Because I havent seen a single modder here, besides Putnam and me, and both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.

I was basically thinking that there was enough feedback to ensure the basic concept worked, and then it would be better to get a working implementation before soliciting content.

The format is designed so that any simple / raw only mod - yeah, MW is unlikely to work - can just be dropped in.  Mods which go deliberately minimalist will obviously be more widely compatible for the merge stage, but there's plenty that works well enough for testing. 

[PeridexisErrant]

I've substantially refactored the code.  We now have one monster procedure that simplifies folders (deletes all files that aren't needed per our mod format), some placeholder input stuff to get an ordered list of mods to load, and then for each mod pass each filename to a merge function for one-file-at-a-time merging. 

So basically that function needs a lot more work, but the rest is at least good enough to work as a placeholder. 

Code: [Select]

def per_file_mod_merge_logic(vanilla_raw_folder, mod_raw_folder, mixed_raw_folder, file):
    if os.path.isfile(mixed_raw_folder + file):
        # preprocess files here
        pass
        # merge logic goes here
        # see https://docs.python.org/2/library/difflib.html
    else:
        pass
        # nothing for now, but later just copy file over:
        #shutil.copy(mod_raw_folder + file, mixed_raw_folder)
https://github.com/PeridexisErrant/Py-Mod-Loader

I had a look at the functions King Mir wrote earlier in the thread, but just got errors.  Do you mind having another look, and maybe trying to make them usable in the function above?  Once we get that working, we have a working prototype!

Spoiler: King Mir's Code (click to show/hide)

I wrote up a simple python function that tests if merging is possible. I test it by passing strings, making it diff on individual letters, but if you pass it a list of strings, it will diff that as lines.

Code: [Select]

import difflib

def can_merge_seq (mod_text, vanilla_text, into_text):
    van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)   
    van_gen_match = difflib.SequenceMatcher(None, vanilla_text, into_text)   
   
    van_mod_seqs = van_mod_match.get_matching_blocks()
    van_gen_seqs = van_gen_match.get_matching_blocks()
   
    cur_v = 0
    while cur_v < len(vanilla_text) :
        (i1,j1,n1) = van_mod_seqs[0]
        (i2,j2,n2) = van_gen_seqs[0]
        if i1 > cur_v and i2 > cur_v:
            return False
        if i1 + n1 - cur_v < i2 + n2  - cur_v:
            cur_v = i1+n1
            van_mod_seqs.pop(0)
        else:
            cur_v = i2 + n2
            van_gen_seqs.pop(0)       
    return True


Here's the code to merge mod files in Python. It merges non-conflicting sequences as proven by the previously published can_merge_seq function.

Neither of the functions can handle two mods adding the same content yet. Also, this still needs to be hooked up to PeridexisErrant's code.

Code: [Select]

import difflib

def do_merge_seq (mod_text, vanilla_text, gen_text):
    if vanilla_text == gen_text: #this should happen often
        return mod_text
    van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)   
    van_gen_match = difflib.SequenceMatcher(None, vanilla_text, gen_text)   
   
    van_mod_ops = van_mod_match.get_opcodes()
    van_gen_ops = van_gen_match.get_opcodes()
   
    output_file_temp = []   
    cur_v = 0
    while cur_v < len(vanilla_text) :
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        #print van_mod_ops[0]
        #print van_gen_ops[0]
        #print cur_v
        if mod_tag == 'equal' and gen_tag == 'equal' :
            if mod_i2 < gen_i2:
                output_file_temp += vanilla_text[cur_v:mod_i2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
            else:
                output_file_temp += vanilla_text[cur_v:gen_i2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)           
        else:                                                 
            if mod_tag != 'equal' :
                output_file_temp += mod_text[mod_j1:mod_j2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_gen_ops.pop(0)   
            elif gen_tag!='equal':
                output_file_temp += gen_text[gen_j1:gen_j2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)   
            #if neither gen_tag nor mod_tag is 'equal', this mod can't be merged
        #print (output_file_temp)
    if van_mod_ops:
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        output_file_temp += mod_text[mod_j1:mod_j2]
    if van_gen_ops:
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        output_file_temp += gen_text[gen_j1:gen_j2]
    return output_file_temp


I'm also not quite sure what the final position on preprocessing of the files was - we have so many test snippets around I can't tell which would be good to use.  Thistleknot - could you post a description of how you think files should be processed before doing the diffs?

[Merkator]

We have still problems with diff. Maybe use something like little extension to diff format, in that way the script can find correct line numbers and fill them. That way we can easily generate correct diff file without much additional work.
I think using git for mod management is overkill. KISS rule win ;P.

My idea is use something like that:

Code: [Select]


Standart diff format look like this

@@ -4,6 +4,6 @@
- something
+ something else
...

We can use something like that

@@ =[CREATURE:WOLVERINE]>[CASTE:MALE], 6 =[CREATURE:WOLVERINE]>[CASTE:MALE],6
-something
+something else
...


The stuff after equality sign describe where the mod will be placed.
The whole expression can be read as:
First [CASTE:MALE] token after [CREATURE:WOLVERINE] token.
And what is important modder don't need to even see them.
The script can find those marks and fill them with correct line numbers.

PROS:

• This just work
• Portable
• Don't need any external app. Just tiny (<100 LOC) script.

CONS:

• Still not solve problems with merging two or more raws. Only make merging much simpler
• I don't know if this is big problem, but my approach need special case when mod add raws at the end of file. Something like =END should sove this.

[King Mir]

I've substantially refactored the code.  We now have one monster procedure that simplifies folders (deletes all files that aren't needed per our mod format), some placeholder input stuff to get an ordered list of mods to load, and then for each mod pass each filename to a merge function for one-file-at-a-time merging. 

So basically that function needs a lot more work, but the rest is at least good enough to work as a placeholder. 

Code: [Select]

def per_file_mod_merge_logic(vanilla_raw_folder, mod_raw_folder, mixed_raw_folder, file):
    if os.path.isfile(mixed_raw_folder + file):
        # preprocess files here
        pass
        # merge logic goes here
        # see https://docs.python.org/2/library/difflib.html
    else:
        pass
        # nothing for now, but later just copy file over:
        #shutil.copy(mod_raw_folder + file, mixed_raw_folder)
https://github.com/PeridexisErrant/Py-Mod-Loader

I had a look at the functions King Mir wrote earlier in the thread, but just got errors.  Do you mind having another look, and maybe trying to make them usable in the function above?  Once we get that working, we have a working prototype!

Spoiler: King Mir's Code (click to show/hide)

I wrote up a simple python function that tests if merging is possible. I test it by passing strings, making it diff on individual letters, but if you pass it a list of strings, it will diff that as lines.

Code: [Select]

import difflib

def can_merge_seq (mod_text, vanilla_text, into_text):
    van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)   
    van_gen_match = difflib.SequenceMatcher(None, vanilla_text, into_text)   
   
    van_mod_seqs = van_mod_match.get_matching_blocks()
    van_gen_seqs = van_gen_match.get_matching_blocks()
   
    cur_v = 0
    while cur_v < len(vanilla_text) :
        (i1,j1,n1) = van_mod_seqs[0]
        (i2,j2,n2) = van_gen_seqs[0]
        if i1 > cur_v and i2 > cur_v:
            return False
        if i1 + n1 - cur_v < i2 + n2  - cur_v:
            cur_v = i1+n1
            van_mod_seqs.pop(0)
        else:
            cur_v = i2 + n2
            van_gen_seqs.pop(0)       
    return True


Here's the code to merge mod files in Python. It merges non-conflicting sequences as proven by the previously published can_merge_seq function.

Neither of the functions can handle two mods adding the same content yet. Also, this still needs to be hooked up to PeridexisErrant's code.

Code: [Select]

import difflib

def do_merge_seq (mod_text, vanilla_text, gen_text):
    if vanilla_text == gen_text: #this should happen often
        return mod_text
    van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)   
    van_gen_match = difflib.SequenceMatcher(None, vanilla_text, gen_text)   
   
    van_mod_ops = van_mod_match.get_opcodes()
    van_gen_ops = van_gen_match.get_opcodes()
   
    output_file_temp = []   
    cur_v = 0
    while cur_v < len(vanilla_text) :
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        #print van_mod_ops[0]
        #print van_gen_ops[0]
        #print cur_v
        if mod_tag == 'equal' and gen_tag == 'equal' :
            if mod_i2 < gen_i2:
                output_file_temp += vanilla_text[cur_v:mod_i2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
            else:
                output_file_temp += vanilla_text[cur_v:gen_i2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)           
        else:                                                 
            if mod_tag != 'equal' :
                output_file_temp += mod_text[mod_j1:mod_j2]
                cur_v = mod_i2
                van_mod_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_gen_ops.pop(0)   
            elif gen_tag!='equal':
                output_file_temp += gen_text[gen_j1:gen_j2]
                cur_v = gen_i2
                van_gen_ops.pop(0)
                if mod_i2 == gen_i2 :
                    van_mod_ops.pop(0)   
            #if neither gen_tag nor mod_tag is 'equal', this mod can't be merged
        #print (output_file_temp)
    if van_mod_ops:
        (mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
        output_file_temp += mod_text[mod_j1:mod_j2]
    if van_gen_ops:
        (gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
        output_file_temp += gen_text[gen_j1:gen_j2]
    return output_file_temp


Yep, I was planning to do that already.

Code: [Select]

I'm also not quite sure what the final position on preprocessing of the files was - we have so many test snippets around I can't tell which would be good to use.  Thistleknot - could you post a description of how you think files should be processed before doing the diffs?
[/quote]My position is avoid preprocessing that changes the mod itself, but do do preprocessing during integration.

As for what's implemented, I think thistleknot figured out how to flatten raws, but it's in sh shell scripts/stray shell commands. But apparently is was helpful in updating his mods to 0.40 so we have a useful proof of concept. And you can lift the regular expressions from it.

[thistleknot]

I would look into diff3 vs diff.  It might be a better way to merge.  If deprecated items are modded in 1st using diff3, and diff3 is always referring to a common ancestor, I think one might get better workable mods.  I tried it out with various text files as input and it does a pretty good job of cascading deletions.  It's just subtractive mods [I think] have to be applied first.

Code: [Select]

dir /b %2\raw\objects
mkdir VersionABC
mkdir VersionABC\raw
mkdir VersionABC\raw\objects
for /f %%f in ('dir /b %2\raw\objects') do diff3 --merge %1\raw\objects\%%f %2\raw\objects\%%f %3\raw\objects\%%f > VersionABC\raw\objects\%%f

Here's a working windows script

Code: [Select]

diff3Batch.bat accmod 34_11 civforge

linux psuedo that I derived it from

Code: [Select]

mkdir VersionABC
for i in (cd VersionA && ls -R); do
  if [ -f VersionA/$i ]; then
    diff3 --merge VersionB/$i VersionA/$i VersionC/$i > VersionABC/$i
  else
    mkdir -p VersionABC/$i
  fi
done

makes a new versionABC.

The format of diff3 is MyChanges Base TheirChanges

So...

MyChanges would be the parameter we keep swapping with each new rebuild

Base would be vanilla

TheirChanges would be the incoming mod.

I tested it out.

Automerges the text files in object

Spoiler: entity_default.txt snippet (click to show/hide)

Code: [Select]

[ARMOR:ITEM_ARMOR_CLOAK:COMMON]
<<<<<<< accmod\raw\objects\entity_default.txt
||||||| 34_11\raw\objects\entity_default.txt
[ARMOR:ITEM_ARMOR_TUNIC:COMMON]
[ARMOR:ITEM_ARMOR_TOGA:UNCOMMON]
[ARMOR:ITEM_ARMOR_VEST:COMMON]
[ARMOR:ITEM_ARMOR_DRESS:COMMON]
[ARMOR:ITEM_ARMOR_ROBE:COMMON]
=======
[ARMOR:ITEM_ARMOR_TUNIC:COMMON]
[ARMOR:ITEM_ARMOR_TOGA:UNCOMMON]
[ARMOR:ITEM_ARMOR_VEST:COMMON]
[ARMOR:ITEM_ARMOR_DRESS:COMMON]
[ARMOR:ITEM_ARMOR_ROBE:COMMON]
[ARMOR:ITEM_ARMOR_APRON:COMMON]
>>>>>>> civforge\raw\objects\entity_default.txt
[HELM:ITEM_HELM_HELM:COMMON]
[HELM:ITEM_HELM_HOOD:COMMON]
[HELM:ITEM_HELM_COIF:COMMON]
[GLOVES:ITEM_GLOVES_GAUNTLETS:COMMON]
<<<<<<< accmod\raw\objects\entity_default.txt
||||||| 34_11\raw\objects\entity_default.txt
[GLOVES:ITEM_GLOVES_GLOVES:COMMON]
[GLOVES:ITEM_GLOVES_MITTENS:COMMON]
=======
[GLOVES:ITEM_GLOVES_CHAIN:COMMON]
[GLOVES:ITEM_GLOVES_SCALE:UNCOMMON]
[GLOVES:ITEM_GLOVES_LAMELLAR:RARE]
[GLOVES:ITEM_GLOVES_GLOVES:COMMON]
[GLOVES:ITEM_GLOVES_MITTENS:COMMON]
>>>>>>> civforge\raw\objects\entity_default.txt
[SHOES:ITEM_SHOES_SHOES:COMMON]
[SHOES:ITEM_SHOES_BOOTS:COMMON]

Update:
Apparently that won't work... Duplicates

[thistleknot]

okay, I think i figured out what's up with diff3

I needed the -e or -a option.

Then I didn't get any output with <<<<< or ====

Code: [Select]

C:\temp>DIFF3 -e -m civforge\raw\objectS\entity_default.txt 34_11\raw\objects\en
tity_default.txt accmod\raw\objects\entity_default.txt >temp.txt

updated batch script

Code: [Select]

dir /b %2\raw\objects
mkdir VersionABC
mkdir VersionABC\raw
mkdir VersionABC\raw\objects
for /f %%f in ('dir /b %2\raw\objects') do diff3 -e --merge %1\raw\objects\%%f %2\raw\objects\%%f %3\raw\objects\%%f > VersionABC\raw\objects\%%f


I think correct order is

Code: [Select]

diff3batch modb moda modc
moda:modc changes will be merged into modb, moda serves as base.

Here's a relevant section and how it was handled by the merge, I think it merge wonderfully.

Left = 34_11; Middle = AccMod; Right = CivForge

You can see it respected the 34_11 - AccMod entity changes when it looked at Civilization Forge, and it added in an item from Civilization Forge, the jar.

So... diff3 diff3 diff3!
http://imgur.com/SOitufk

Note: It looks like the behavior dropped toy_doll from being added by CivForge due to the fact that the entire section was deleted by Accelerated mod.

The notepad++ window is the output of the diff3 -e -m merge operation.  Behind it, the center column in kdiff3 looks just like the column on the left.  The only difference is the right column (civforge) has a _Jar entry that is added in the final output.

Update:
I was able to merge 3 34_11 mods together and then update them to 40_09 using diff3.  I went through and randomly verified some of the additions that you can see were brought in from my github version tracking.  For example, the changes from 34_11:Fortress Defense entity_default can be seen here (although similar to accelerated mod, Fortress Defense entity_default optional changes also got rid of wheelbarrow's for plainsmen, which accelerated mod did not, and those changes were merged in) 

https://github.com/thistleknot/df_34_11_win-Flattened-Bases/commit/8d71025dfc2b8996245f2b0ff4ee469b747233a8

I merged
34_11:Accelerated changes into CivForge = AccCivForge
34_11:Fortress Defense changes into AccCivForge = AccCivForgeFortressDefense
34_11:40_09 changes into AccCivForgeFortressDefense = AccCivForgeFortressDefense-40_09

The only caveat, is this batch file only works on the files in raw\objects.  Diff3 can be modified to work with other files.

Also, the batch file only looks at the [directory listing of] base folder's raw\objects\*.* for diff3 comparisons.  New files that are not in this folder due to other mods having them, won't have a diff3 merge ran on them because their absent from this initial directory listing (as a workaround, an empty file could be created for those missing files, diff3 should create a new outputted file for each missing file if they are supplied in this manner).  So if two mods mod the same file but the base doesn't have it.  Then a 2 way merge would have to be done between the two outside of this diff3 batch file and then that file would have to be brought into the final output.

On a final note, alphabetizing:
The only thing stopping us from doing something like this EFFICIENTLY on a total conversion mod... is a [command line] raw/text object alphabetizer.  Maybe with extended functionality to alphabetize castes within object:id's.

However.  If one wanted to incorporate a total conversion mod.  One could... alphabetize all the raws by hand using rawExplorer.  It can alphabetize a whole set of raws apparently and output it in a single go, then we parse them.  Then do the same with vanilla base.  And you could probably do better additive/subtractive diff3 comparisons.