Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[hermes]

No, it could still be a problem.  Imagine vanilla has raws for A, B, and C.  Mod1 modifies A and removes C.  Mod2 doesn't modify A or C - if it did we would refuse the merge - but instead modifies B, which has C as a dependency.  The raws are now broken, unless we have some *really* impressive parsing to catch this kind of thing. 

I don't know how common this might be, but for minor mods it doesn't seem too likely.  I think the best way to deal with this in the short to medium term is just to live with it and reduce the chances by way of major mods, which are more likely to do this, generally being incompatible.

Agreed.  I was being rather overly negative, the blind patch diff will work great on merging mods that don't remove anything, and will catch more problems than simply checking for file overwrites.

The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things.  Maybe it just needs to be the full raw folder instead of files changed from vanilla...

That shouldn't be such a big problem as the modder/an overseer could just put a blank file in instead..?

[PeridexisErrant]

OK, here's where I leave this for while:  input format, folder structure, and a code skeleton that works... for everything except the actual merge bit.  You may want to expand the information passed through the functions to the merge logic, I kept it minimal so it doesn't seem to even hold folder variables.

Download link.

Code: (just the mod merging functions) [Select]

def make_raws_with_mods():
    print('What mods do you want to load?')
    for mod in mod_folders_list:
        print('   ', mod_folders_list.index(mod), mod)
    mods_to_load = input('Enter the indicies of the mods to load, in order, seperated by spaces:\n    ')
    mods_to_load = mods_to_load.split(' ')
    mod_load_order = []
    for index in mods_to_load:
        mod_load_order.append(mod_folders_list[int(index)])
    mixed_raws_folder = mods_folder + 'temp/raw/'
    # remove an old folder if exists.  Name reserved for this reason!
    if os.path.exists(mixed_raws_folder):
        if os.path.isfile(mixed_raws_folder):
            os.remove(mixed_raws_folder)
        else:
            shutil.rmtree(mixed_raws_folder)
    # create folder of vanilla raws to operate on
    shutil.copytree(vanilla_raw_folder, mixed_raws_folder)
    print('\nFolder for merging created - "'+mixed_raws_folder+'" - with vanilla raws.')
    # start merging mods!
    merge_next_mod('', mod_load_order, -1)
    # get back after looping through
    print('\nMod merging complete!  The merged mods can be found in the mod folder as "temp".')

def merge_next_mod(next_mod, mod_load_order, mods_merged):
    if not next_mod == '':
        mod_merge_logic(next_mod, mods_merged)
    if not mod_load_order == []:
        next_mod = mod_load_order.pop(0)
        mods_merged += 1
        merge_next_mod(next_mod, mod_load_order, mods_merged)

def mod_merge_logic(mod, mods_merged):
    print('\n(placeholder merge logic for', mod + '; no action taken;', mods_merged, 'mods merged already)')
    # see eg vanilla file removal function to do per-file comparisons

Spoiler: all code (click to show/hide)

import os
import shutil
import filecmp

# path from script to mods folder
mods_folder = 'LNP/Mods/'

mod_folders_list = []
for mod in os.listdir(mods_folder):
    if mod.startswith('df_'):
        # there must be only one folder beginning 'df_', the basis of comparison
        vanilla_folder = mods_folder + mod
        vanilla_raw_folder = vanilla_folder + '/raw/'
    elif mod == 'temp':
        pass
    else:
        mod_folders_list.append(mod)

def simplify_mod_and_df_folders():
    for mod in os.listdir(mods_folder):
        files_removed = 0
        folders_removed = 0
        for item in os.listdir(mods_folder + mod):
            # delete anything top-level not containing string 'raw', 'readme', or 'config'
            if item == 'raw':
                pass
            elif 'readme' in item.lower():
                pass
            elif 'config' in item.lower():
                pass
            elif os.path.isfile(mods_folder + mod + '/' + item):
                os.remove(mods_folder + mod + '/' + item)
                files_removed += 1
            else:
                shutil.rmtree(mods_folder + mod + '/' + item)
                folders_removed += 1
        if not files_removed + folders_removed == 0:
            print(mod, 'folder simplified!  (removed', files_removed, 'files and', folders_removed, 'folders)')

def remove_vanilla_files_from_mod_raws_and_add_blanks_to_preserve_deletions_in_mod():
    for mod in mod_folders_list:
        mod_raw_folder = mods_folder + mod + '/raw/'
        files_removed = 0
        blank_files_added = 0
        # add a blank file where the mod does not have a vanilla file
        for file_tuple in os.walk(vanilla_raw_folder):
            for item in file_tuple[2]:
                item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(vanilla_raw_folder, '')
                if not os.path.isfile(mod_raw_folder + item_path_str):
                    open(mod_raw_folder + item_path_str, a).close()
                    blank_files_added += 1
        # remove files identical to vanilla raws
        for file_tuple in os.walk(mod_raw_folder):
            for item in file_tuple[2]:
                item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(mod_raw_folder, '')
                if os.path.isfile(vanilla_raw_folder + item_path_str): # if the file exists in the vanilla raws
                    if filecmp.cmp(mod_raw_folder + item_path_str, vanilla_raw_folder + item_path_str): # and it's the same file
                        os.remove(mod_raw_folder + item_path_str)
                        files_removed += 1
        if blank_files_added > 0:
            print('\n' + mod + ':\n    removed', blank_files_added, 'to preserve files removed compared to vanilla raws')
        if files_removed > 0:
            print('\n' + mod + ':\n    removed', files_removed, 'files identical to vanilla raws')

def make_raws_with_mods():
    print('What mods do you want to load?')
    for mod in mod_folders_list:
        print('   ', mod_folders_list.index(mod), mod)
    mods_to_load = input('Enter the indicies of the mods to load, in order, seperated by spaces:\n    ')
    mods_to_load = mods_to_load.split(' ')
    mod_load_order = []
    for index in mods_to_load:
        mod_load_order.append(mod_folders_list[int(index)])
    mixed_raws_folder = mods_folder + 'temp/raw/'
    # remove an old folder if exists.  Name reserved for this reason!
    if os.path.exists(mixed_raws_folder):
        if os.path.isfile(mixed_raws_folder):
            os.remove(mixed_raws_folder)
        else:
            shutil.rmtree(mixed_raws_folder)
    # create folder of vanilla raws to operate on
    shutil.copytree(vanilla_raw_folder, mixed_raws_folder)
    print('\nFolder for merging created - "'+mixed_raws_folder+'" - with vanilla raws.')
    # start merging mods!
    merge_next_mod('', mod_load_order, -1)
    # get back after looping through
    print('\nMod merging complete!  The merged mods can be found in the mod folder as "temp".')

def merge_next_mod(next_mod, mod_load_order, mods_merged):
    if not next_mod == '':
        mod_merge_logic(next_mod, mods_merged)
    if not mod_load_order == []:
        next_mod = mod_load_order.pop(0)
        mods_merged += 1
        merge_next_mod(next_mod, mod_load_order, mods_merged)

def mod_merge_logic(mod, mods_merged):
    print('\n(placeholder merge logic for', mod + '; no action taken;', mods_merged, 'mods merged already)')
    # see eg vanilla file removal function to do per-file comparisons

simplify_mod_and_df_folders()
remove_vanilla_files_from_mod_raws_and_add_blanks_to_preserve_deletions_in_mod()
make_raws_with_mods()

Code: [Select]

The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things.  Maybe it just needs to be the full raw folder instead of files changed from vanilla...

That shouldn't be such a big problem as the modder/an overseer could just put a blank file in instead..?

That is the obvious workaround, and if we do it in the code modders don't have to comply with the standard at all.  Just reverse the current comparison tool to compare "for each file in vanilla, if no such file in mod, create blank file by that name".  Identical files are still removed which saves a lot of space for micro mods, and the diff from vanilla to an empty file is tiny so no worries there either.  Included above.

[King Mir]

The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things.  Maybe it just needs to be the full raw folder instead of files changed from vanilla...

That shouldn't be such a big problem as the modder/an overseer could just put a blank file in instead..?

That is the obvious workaround, and if we do it in the code modders don't have to comply with the standard at all.  Just reverse the current comparison tool to compare "for each file in vanilla, if no such file in mod, create blank file by that name".  Identical files are still removed which saves a lot of space for micro mods, and the diff from vanilla to an empty file is tiny so no worries there either.  Included above.

You could do this as a setting, but for minor mods, I think they're likely to just be the modified file in the proper raw folder and nothing else. So the default should be no file implies no changes.

[thistleknot]

No, it could still be a problem.  Imagine vanilla has raws for A, B, and C.  Mod1 modifies A and removes C.  Mod2 doesn't modify A or C - if it did we would refuse the merge - but instead modifies B, which has C as a dependency.  The raws are now broken, unless we have some *really* impressive parsing to catch this kind of thing. 

I don't know how common this might be, but for minor mods it doesn't seem too likely.  I think the best way to deal with this in the short to medium term is just to live with it and reduce the chances by way of major mods, which are more likely to do this, generally being incompatible. 

The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things.  Maybe it just needs to be the full raw folder instead of files changed from vanilla...

so you guys are going to try "ala carte adding".  I thought you would have static 1 time patches only.  Not multiple.  If your doing multiple, your going to need some sort of object tracking as stated beforehand.  However... that could be something done at a later time.

[PeridexisErrant]

For clarity:

 - in the format, a missing file means "this file is identical to vanilla". Which means that minor mods can consist of a single file in the proper folder.

 - some mods actually do delete vanilla files though - so to cover that use case, files that are present in vanilla but need to be deleted in the mod are represented by an empty text file (equivalent to no file to DF).

 -  the code I've posted above will correctly convert full mod installs in the folder structure given in the download into our specified format, including blank placeholder file creation.

It also includes a first stab at the structure of a mod merge tool, albeit one utterly without logic (it prints the name of each mid and nothing else). I'm hoping some of you can code something in the function I defined for merge logic.

I need to read up again on how passing arguments to functions works, in order to keep all the useful variables around. $var not defined errors I didn't have time to deal with put an end to my merge logic creation for the day, so I just posted the code I had.

A la carte mod merging is obviously risky in some cases, but in others - like merging minimal minor mods - it would be an awesome trick. Starting simple and then expanding, with our easy to make and use format as a base, also leaves us the potential to keep improving the logic. At this point we've seen enough interest that I think we might see a snowball of usage and improvements, which would be awesome. Loading just one mod without graphics isn't a big enough trick to really count as moving forward though, and that's all I've got so far. Soon...

[Button]

... Which is exactly why I'd go the route of Valdemar's stuff.  First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis.  It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on.  You have two levels of differences - the objects present difference, and the difference between equatable objects themselves.  The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".

In case 1 you'd know that the added lines of B would at least be in the correct object, and if the program was smart it could drop those lines in right after the correct preceding line.

In case 2 you'd know in what object the conflict arose and could prompt the user with relevant information.

This way would make this basic stuff easier, and open the door for more advanced stuff like making connections throughout the raws to entirely remove problem objects.  The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.

The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner.  PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.

I'm with Hermes.

IMO a tool that can only load a single minor mod is trivial, because most minor mods are set up to just overwrite the vanilla folder. If you want a mod loader that can load one mod at a time, it doesn't need all this logic - just a vanilla raw directory, a bunch of minor mod raw directories, and the ability to select which minor mod you want to load on top of the vanilla raw directory in the application raw folder. To load one mod, there's no need to diff. So I assume you want something that can load more than one set of raws... but merging multiple mods through diff would be a nightmare. The dupes alone... Not to mention the really problematic part of merging, creature variations.

For example, the vampires-drink-booze-fix I have uses creature variations to replace humanoids' blood with BLOOD2 or ICHOR2, which is default blood with a syndrome. Diff wouldn't find any problems loading the boozefix alongside another mod that did something to blood, but trying to load a creature with modifications to BLOOD after BLOOD has already been replaced with BLOOD2 (and so no longer exists in the creature definition)? There's no way to find those kinds of conflicts without loading raws.

[Dirst]

... Which is exactly why I'd go the route of Valdemar's stuff.  First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis.  It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on.  You have two levels of differences - the objects present difference, and the difference between equatable objects themselves.  The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".

In case 1 you'd know that the added lines of B would at least be in the correct object, and if the program was smart it could drop those lines in right after the correct preceding line.

In case 2 you'd know in what object the conflict arose and could prompt the user with relevant information.

This way would make this basic stuff easier, and open the door for more advanced stuff like making connections throughout the raws to entirely remove problem objects.  The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.

The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner.  PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.

I'm with Hermes.

IMO a tool that can only load a single minor mod is trivial, because most minor mods are set up to just overwrite the vanilla folder. If you want a mod loader that can load one mod at a time, it doesn't need all this logic - just a vanilla raw directory, a bunch of minor mod raw directories, and the ability to select which minor mod you want to load on top of the vanilla raw directory in the application raw folder. To load one mod, there's no need to diff. So I assume you want something that can load more than one set of raws... but merging multiple mods through diff would be a nightmare. The dupes alone... Not to mention the really problematic part of merging, creature variations.

For example, the vampires-drink-booze-fix I have uses creature variations to replace humanoids' blood with BLOOD2 or ICHOR2, which is default blood with a syndrome. Diff wouldn't find any problems loading the boozefix alongside another mod that did something to blood, but trying to load a creature with modifications to BLOOD after BLOOD has already been replaced with BLOOD2 (and so no longer exists in the creature definition)? There's no way to find those kinds of conflicts without loading raws.

Thistleknot is working on a more robust merge tool, and there's also Rubble.

PeridexisErrant is aiming for a simple way to sample mods, and for a pre-1.0 it's perfectly fine for it to be one mod at a time.  I believe that a useable tool should be able to handle some simple merging, like putting two mod's worth of PERMITTED_REACTIONs into the same entity_default.txt file, or modifying one creature when a creature before it in the same file had a line added/removed.  In this regime, two mods are in conflict only in the narrow sense that they attempt to modify the same vanilla tag.  That could be an insta-fail or a last-in-wins depending on settings, especially since load order is user-definable.

A manifest file (which is planned) can list dependencies, and it could also list known semantic conflicts like the one you mentioned.  Such a manifest entry could specify whether the mod is completely incompatible, or there is a required load order.  That presents comprehensible guidance to the player while minimizing the complexity (i.e., no recursive searches through the raws to locate unused tissues).

[Button]

PeridexisErrant is aiming for a simple way to sample mods, and for a pre-1.0 it's perfectly fine for it to be one mod at a time.  I believe that a useable tool should be able to handle some simple merging, like putting two mod's worth of PERMITTED_REACTIONs into the same entity_default.txt file, or modifying one creature when a creature before it in the same file had a line added/removed.  In this regime, two mods are in conflict only in the narrow sense that they attempt to modify the same vanilla tag.  That could be an insta-fail or a last-in-wins depending on settings, especially since load order is user-definable.

But the idea is to make it newb-friendly, which means that it needs to be able to identify potential problems and handle them gracefully:

if the logic cannot produce a known correct result, the merge should be refused.  This protects users from broken raws; put another way it's the difference between a mod loader which does some merging and a mod merge tool.

False negatives are OK from this perspective - just limiting what mods you can throw together - but false positives are a huge potential user experience problem, especially for a newbie. My position is that there are just too many conflicts that are invisible to a file-by-file diff to rule out false positives unless you first manually screen out any mods which are potentially problematic.

A manifest file (which is planned) can list dependencies, and it could also list known semantic conflicts like the one you mentioned.  Such a manifest entry could specify whether the mod is completely incompatible, or there is a required load order.

It's planned? Haven't seen it on this thread. As far as manifest files, PE said just the other day that:

I am strongly opposed to anything which would require a more complicated input than folders of changed raws, as outlined in OP.  This is because I feel having a simple, usable, standard format is very important for adoption from both users and modders.  If it becomes common to have a readme.txt and configuration.xml - to hold info such as author, name, homepage, base DF version, etc in the latter case - it would be good to use that information but their absence must be handled gracefully.

Which would seem to imply no manifest file required. If planning is going on/that requirement has been reversed elsewhere, I'll shut up and let you work, since I don't want to butt in when I don't have all the information.

[thistleknot]

Updated:

Code: [Select]

sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" testfile.txt
with batch

Still needs a little work:
broken token's have odd behaviour

Code: [Select]

echo off
for /f %%a in ('dir /b *.txt') do sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" %%~na.txt > %%~na.out
REM erase *.txt
REM ren *.out *.txt
REM remove all blanklines
for /f %%a in ('dir /b *.out') do sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" %%~na.out > %%~na.out2
echo on

input:

Code: [Select]

item_gloves

[OBJECT:ITEM]

###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]
###test###[ARMORLEVEL:2]
[UPSTEP:1]
###test###[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]


output:

Code: [Select]

item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]



[Dirst]

A manifest file (which is planned) can list dependencies, and it could also list known semantic conflicts like the one you mentioned.  Such a manifest entry could specify whether the mod is completely incompatible, or there is a required load order.

It's planned? Haven't seen it on this thread. As far as manifest files, PE said just the other day that:

I am strongly opposed to anything which would require a more complicated input than folders of changed raws, as outlined in OP.  This is because I feel having a simple, usable, standard format is very important for adoption from both users and modders.  If it becomes common to have a readme.txt and configuration.xml - to hold info such as author, name, homepage, base DF version, etc in the latter case - it would be good to use that information but their absence must be handled gracefully.

Which would seem to imply no manifest file required. If planning is going on/that requirement has been reversed elsewhere, I'll shut up and let you work, since I don't want to butt in when I don't have all the information.

A manifest is the third point on the OP, though I did overstate it a bit by implying it was a definite feature when it's just being looked at.

I do differ with PE a bit because I think asking the modder to document known incompatibilities isn't much of a burden.  If we get around to a stretch goal of using that manifest file to check for mod updates, it will ripple out to the users with minimal effort.  The thorny issue is what to do if the tool discovers an active game has a newly discovered conflict.

Warning:
You are using RampageMod v1.08 and Zootastic v2.0, and an incompatibility between these mods has been identified.

Severity:
Minor

The problem:
Both of these mods affect several mundane animals in different ways.  The combination of mods does not cause an error, but it does affect the game balance that each mod is attempting to achieve.

Recommended actions:
It is recommended that you take one of the following actions.
(1) Ensure that Zootastic is loaded before RampageMod.
(2) Update to RampageMod v1.09.
Unfortunately, none of the above changes can be applied to active game.  We apologize for the inconvenience.

[thistleknot]

finally, here it is.

I found an easier way to extract tokens, went with it.  I didn't have to solve a lot of the other work (you can see it in the REM lines)

Code: [Select]

echo off
REM s/\][^]]*/&\n/g doesn't work right
REM to remove [[ and ]] s/\][^]]*/&\n/g; s/\[[^[]*/&\n/g
REM to split up ][
REM for /f %%a in ('dir /b *.txt') do sed -e "s/\]\[/\]\n\[/g" %%~na.txt > %%~na.out
REM put tokens on their own line
for /f %%a in ('dir /b *.txt') do sed -e "s/\[[^][]*\]/\n&\n/g" %%~na.txt > %%~na.out
REM remove tabs
for /f %%a in ('dir /b *.out') do sed -r "s/\t//g" %%~na.out > %%~na.out2
REM for /f %%a in ('dir /b *.out') do sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" %%~na.out > %%~na.out2
REM remove all blanklines
for /f %%a in ('dir /b *.out2') do sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" %%~na.out2 > %%~na.out3
REM cleanup
REM erase *.txt
REM ren *.out3 *.txt
REM erase *.out
echo on

basically what it does is no matter how bad the token's are broken, it puts them on their own line as a [token].

The rest of the broken comments are left alone an their own lines.

input

Spoiler (click to show/hide)

Code: [Select]

item_gloves

[OBJECT:ITEM]

###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]
###test###[ARMORLEVEL:2][[[]][
[UPSTEP:1][[
###test###[SHAPED]
]][LAYER:ARMOR]###test######test###
[COVERAGE:100]
]][LAYER_SIZE:20]][
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]



output

Spoiler (click to show/hide)

Code: [Select]

item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[[
[]
][
[UPSTEP:1]
[[
###test###
[SHAPED]
]]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
]]
[LAYER_SIZE:20]
][
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]



[PeridexisErrant]

Very nice!  There are a couple of cosmetic issues though; the output contains the input twice over (I suspect a bug), and a final pass to remove any line consisting entirely of square brackets would be nice.  One those are dealt with, I think that looks more human-readable as well as better for diffs. 

The problem with relying on a manifest is that it massively cuts down on your potential input.  At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged.  If we require a manifest, that shrinks the pool to those mods made with this tool in mind or manually updated or compatibility by someone, which is to say no mods at all.  So requiring a manifest is fine, so long as we can derive a sensible one from the mod alone, and the defaults aren't too restrictive.  Good ways to use it would be to show extra information to the user (author, update link, etc), or helpful but non-critical info for the program - eg known non-conflicting mods (useful for mods that were split up, program then ignores detected conflicts between them as false), base raws version, etc.  Again though, the program should work well enough without a manifest file. 

Complicated problems in the raws following multiple merges:  there's probably something possible with comparing the before and after line identifiers in diffs and working out if two are intending to apply to the same area, even if the content at that line has changed.  This would deal with a couple of the problems noted so far.  The order might be:
 - transform all mod folders into our format, permanently (once ever per mod)
 - create (temporary) flattened raws for vanilla and all mods (once per run)
 - create a diff between flat vanilla and each flat mod (once per run)
 - select mods to load and load order, and each time this changes:
 * analyse diffs in order; if changed areas overlap reject merge and return first overlapping mod
 - attempt merges in order; if any fail reject the merge and return first non-merging mod
 - if merge was rejected inform the user which mod caused failure, otherwise offer new raws
I think at this point we're mostly talking over how to do the starred point to avoid false-positives in merge-ability.  While I think this would work enough of the time to ignore the edge cases, I don't have enough modding experience to really tell.  Tying for zero false positives (ie merges that should have been refused) if probably futile, because we'd be back to one mod at a time.  I'm still focussing on normal diffs rather than something raw-aware because it's so much faster to build; we can improve the system after it exists. 

And in the end we can add a label to use at own risk, explain that modding can lead to a broken game and while we try it's not perfect, and if the new world they generate is broken - that's what modding is like sometimes!

[thistleknot]

the double type was an accident

as to:
"remove any line consisting entirely of square brackets would be nice"

that was done intentionally.  To keep the contextual information in place (such as commented out options like mw has).  The mess of brackets was due to me purposely putting a bunch in.

This format should work.  It merely extracts [tokens] and put's them on their own line.  Everything else is left to whatever line it was on.

There is a much simpler version of the script that just grabs the [tokens], but you'd have to reinject the filename at the top

Code: [Select]

sed -e "s/\[[^][]*\]/\n&\n/g"

[hermes]

The problem with relying on a manifest is that it massively cuts down on your potential input.  At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged.  If we require a manifest, that shrinks the pool to those mods made with this tool in mind or manually updated or compatibility by someone, which is to say no mods at all.  So requiring a manifest is fine, so long as we can derive a sensible one from the mod alone, and the defaults aren't too restrictive.  Good ways to use it would be to show extra information to the user (author, update link, etc), or helpful but non-critical info for the program - eg known non-conflicting mods (useful for mods that were split up, program then ignores detected conflicts between them as false), base raws version, etc.  Again though, the program should work well enough without a manifest file. 

I understand the sentiment, but not the logic.  Namely, it doesn't matter what system you implement, modders still have to pack responsibly or you must include a manifest.  Users would have to download mods from DFFD into a Mods directory, or the launcher has a browser that automatically lists, downloads and unpacks mods - either way there is an onus on the modder to conform to the raw folder structure, which they currently do not do at all. 

Creating a catch-all formatter that would magically pull out all the relevant files is impossible.  The Wanderer's Friend mod dumps everything within the raw structure, but the user has to select which features they want to install by selectively copying files across.  Only way you can do this properly is to have a sentient/English speaking AI that doesn't mind spending its life reading mod readme.txt files.

Point is, somewhere in the chain a human has to check it over and understand if the mod is compatible with the loader.  This line...

The problem with relying on a manifest is that it massively cuts down on your potential input.  At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged.

... is false.  Some mods, by chance, will work.  Many/most of them won't or will be incorrectly installed and produce unwanted/unintended behaviour.

If I was going to future proof this, I'd bite the bullet and put a manifest in from the start.  Make another program or a webform or something that can generate the manifest easily, or just some guidelines.  Set up some mods yourself this way and then encourage others to use it.  Make sure the manifest has version control, then later you can make it work with DFHack scripts and other things.

Spoiler: Simple Manifest (click to show/hide)

Wouldn't be too much to ask from modders, and would actually create a standard.  Modders would have a better idea of what functions would be required.  The DFFD ID is a little circular, but the best way to ID mods I think.  Would have to put the file up, get the ID, then make the manifest and reupload... maybe there's a better way to do that.

Code: [Select]

[MOD_MANIFEST_VERSION:0.1]

[MOD_NAME:x]
[VERSION:0.1a]
[RELEASE_DATE:1:1:2015]
[DFFD_ID:1234] // DFFD webpage can be got from this
[HOMEPAGE:www.xxx.com] // Don't go to this site, it could be bad
[PRIMARY_AUTHOR:x]
[SECONDARY_AUTHOR:x]  // Contributors
[SECONDARY_AUTHOR:x]
[DERIVATIVE_OF:1234]  // DFFD id of parent project

[REQUIRES_MOD:0000] // DFFD id
[CONFLICT_MOD:1111]
[CONFLICT_MOD:2222]

[INSTALL:1:INSTALL_PROGRAM:src_file:dest_file]
[INSTALL:2:UNPACK_FILE:src_file:dest_folder]
[INSTALL:3:COPY_FILE:raw/objects/abc.txt]
[INSTALL:4:COPY_FILE:raw/objects/magic_interactions.txt]
[INSTALL:5:REMOVE_FILE:raw/objects/vanilla_raw_file.txt]

[UNINSTALL:1:DELETE:file or folder]
[UNINSTALL:2:DELETE:file or folder]
// etc...
[UNINSTALL:10:RESTORE_VANILLA_FILE:raw/objects/vanilla_raw_file.txt]

[Dirst]

the double type was an accident

as to:
"remove any line consisting entirely of square brackets would be nice"

that was done intentionally.  To keep the contextual information in place (such as commented out options like mw has).  The mess of brackets was due to me purposely putting a bunch in.

This format should work.  It merely extracts [tokens] and put's them on their own line.  Everything else is left to whatever line it was on.

There is a much simpler version of the script that just grabs the [tokens], but you'd have to reinject the filename at the top

Code: [Select]

sed -e "s/\[[^][]*\]/\n&\n/g"

Try this as a sed command

s/(.*?)(\[[^][]+\])/\1\n\2\n

and see if that works.  It looks for "something that's not a valid tag" followed by "something that is a valid tag" and inserts newlines after each.  Consecutive valid tags probably put a blank line in between, but a wrapper (or second pass) can weed all of those out.