Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[King Mir]

You're just doing extra work that way. You just need a list of valid files specified as a mod-relative path.

Quite possibly.  However I'd rather have working code to refactor than spend years in design, so...

Understood. I don't wanna tell you how to code.

Let me know if I can help with anything.

[thistleknot]

I figured out what causes the merge conflicts...

It's when I derive a patch from:

34_11 to accelerated

then...

I try to apply it to

40_08

the conflicts arise because my patch file is confused when it expects a 34_11 line and see's a 40_08 line.  Only way to fix that is to incorporate into the patch the 34_11 to 40_08 changes as well as the accelerated patch

so...

to alleviate this...  it's almost like we need to apply a diff patch of 34_11 to 40_08 + the accelerated patch in the same patch file somehow.

[PeridexisErrant]

To be honest, I think it's perfectly reasonable to ask people to update their mods for a major DF update.  The diff should cope with 40_01 to 40_08 raws and associated changes, but 34.11 to 40.01 is a bit much.  For that case, producing a diff against vanilla of the relevant version should let you manually update pretty easily - but it's not my problem for this tool.  The script I've got so far expects all mods to be for the latest minor version anyway, and that won't change until a configuration.xml (or similar) is implemented. 

[PeridexisErrant]

Happily, it looks there are some rather powerful tools for this kind of thing in the Python standard libraries (who am I kidding, of course there are). 

See especially difflib, which looks like it calculates perfect git-style deltas and patch files.  I imagine that could save you a fair bit of clicking, Thistleknot.
filecmp should also make it trivial to isolate the files which are different to the vanilla raws. 

[hermes]

Just a quick question... Valdemar's mod manager looks like it does the comparison in a better way by pulling out all the entities from the raws and comparing them on an individual basis... Since the code is already available, and this way would make resolving conflicts clearer and easier, why don't you guys use this instead of a text comparison that is, in computational terms, relatively meaningless?  I don't see the long term use of a diff patch.

[PeridexisErrant]

Just a quick question... Valdemar's mod manager looks like it does the comparison in a better way by pulling out all the entities from the raws and comparing them on an individual basis... Since the code is already available, and this way would make resolving conflicts clearer and easier, why don't you guys use this instead of a text comparison that is, in computational terms, relatively meaningless?  I don't see the long term use of a diff patch.

There are two main reasons to use diffs now (though I'm just going to get file overwrites working first):
 - fast, easy, standard logic means that it's easy to implement in this and also for anyone else to make their own compatible launcher
 - the goal of this tool is to act as a mod loader, not an advanced mod merge tool

I'm definitely interested in exploring more advanced merging eventually, but what this needs most of all is to get added to something like the LNP where anyone can use it and add content to it. 

[thistleknot]

I plan on using mod managers merge for my own personal project BTW.

It is entirely possible to do the same thing that mod manager does using regular patch files and merely checking the states of objects before and after merges (python anyone?)

My idea of merging 3411 mods into 40_08 was just an example of PORTING mods over and HOW it can be done, but expect to resolve some minor diff conflicts then derive a final 40_08 diff for your project

[PeridexisErrant]

Ok, I now have code that will turn a folder full od modded DF installs into a much smaller folder full of mods conforming with the format at the top of the thread.  That is, 'LNP/Mods/$mod' contains only files with 'readme' or 'config' in the name, and a raw folder containing those files that are not identical to vanilla. 

For Accelerated Modest Mod (40.08), that shrinks it from 10MB to 2MB.  Not bad for such naive code!

Code: [Select]

import os
import shutil
import filecmp

# path from script to mods folder
mods_folder = 'LNP/Mods/'

mod_folders_list = []
for mod in os.listdir(mods_folder):
    if mod.startswith('df_'):
        # there must be only one folder beginning 'df_', the basis of comparison
        vanilla_folder = mods_folder + mod
        vanilla_raw_folder = vanilla_folder + '/raw/'
    else:
        mod_folders_list.append(mod)


def simplify_mod_and_df_folders():
    for mod in os.listdir(mods_folder):
        files_removed = 0
        folders_removed = 0
        for item in os.listdir(mods_folder + mod):
            # delete anything top-level not containing string 'raw', 'readme', or 'config'
            if item == 'raw':
                pass
            elif 'readme' in item.lower():
                pass
            elif 'config' in item.lower():
                pass
            elif os.path.isfile(mods_folder + mod + '/' + item):
                os.remove(mods_folder + mod + '/' + item)
                files_removed += 1
            else:
                shutil.rmtree(mods_folder + mod + '/' + item)
                folders_removed += 1
        if not files_removed + folders_removed == 0:
            print(mod, 'folder simplified!  (removed', files_removed, 'files and', folders_removed, 'folders)')

def remove_vanilla_files_from_mod_raws():
    for mod in mod_folders_list:
        mod_raw_folder = mods_folder + mod + '/raw/'
        files_removed = 0
        for file_tuple in os.walk(mod_raw_folder):
            for item in file_tuple[2]:
                item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(mod_raw_folder, '')
                if os.path.isfile(vanilla_raw_folder + item_path_str): # if the file exists in the vanilla raws
                    if filecmp.cmp(mod_raw_folder + item_path_str, vanilla_raw_folder + item_path_str): # and it's the same file
                        os.remove(mod_raw_folder + item_path_str)
                        files_removed += 1
        if files_removed > 0:
            print('\n' + mod + ':\n    removed', files_removed, 'files identical to vanilla raws')

def make_raws_with_mods():
    print('What mods do you want to load?')
    for mod in mod_folders_list:
        print('    ', mod_folders_list.index(mod), mod)
    mods_to_load = input('Enter the indicies in load order seperated by spaces.\n    ')

    # gives ordered list with indicies as strings
    mods_to_load = mods_to_load.split(' ')

    print('Just kidding, this doesn\'t do anything yet!')


simple_folders_q = input('Do you want to simplify mod folders? (y/n)\n    ')
if simple_folders_q == 'y':
    simplify_mod_and_df_folders()
    print('  Done!\n')
else:
    print('Not simplifying mod folders\n')

remove_files_q = input('Do you want to remove files from mods which are identical to vanilla raws? (y/n)\n    ')
print(remove_files_q)
if remove_files_q == 'y':
    remove_vanilla_files_from_mod_raws()
    print('  Done!\n')
else:
    print('Not removing vanilla files\n')

make_raws_with_mods()


[thistleknot]

So... I think I may have a solution with how to apply older mods to newer versions of the game.

http://cyberelk.net/tim/patchutils/man/rn01re02.html

http://en.wikipedia.org/wiki/Quilt_%28software%29

Basicaly, merge patch files.

Vs trying to apply a 34_11 mod to 40_08 as a patch of changes that we based on 34_11 vs 40_08...

One could take a diff patch of 34_11 to 40_08,
  call this 34-11To40-08.patch
And a 34_11 to Accelerated,
  call this 34_11ToAccelerated.patch

Then, using the tool combinediff (I guess any conflicts would have to be addressed on the patch file level)

something like combinediff 34-11To40-08.patch 34_11ToAccelerated.patch > 34_11To40_08_Accelerated.patch

and apply this patch to a 34_11 version of the game to achieve a v40_08 mod of the game, then do a diff comparison from 40_08 to this mod of v40_08

In light of what that script is doing, I recommend this one now

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
        $out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"

        Get-Content $file | Foreach-object {
            $_ -replace "`t","" `
-replace '\]\[',"]`r`n["
        } | Set-Content $outFile
    }

I was reading on what it does, and it rewrites the replacement operations.  I'm gonna push a better script in the next day or two if I don't learn a better way to write a script.

[thistleknot]

Touching on Fricy's comment from the link on the 1st page

The best'd be if there were total conversions to choose from, and also mini mods/addon packs that expand the base game, like streamlined leather and display case for eg.

Total conversion almost always entail a full rewrite.  A patch file CAN handle it, but it's merely going to copy/replace existing data.  Small mods will hardly be noticeable, but mods that standardize the raw file say... alphabetically, and rename reactions... I think are going to have huge diff differences.

Btw.  I just realized a minor issue with that powershell script again... I described it in the original stackexchange thread.
http://stackoverflow.com/questions/25345739/powershell-regexp-to-replace-any-character-left-bracket-and-replace-wit

Basically it's not fully putting comments onto their own lines.  But to be honest, anyone who knows python and regular expressions can probably handle it a little better than myself.

[King Mir]

You can just use the 3 regular expressions I wrote. You do have to make sure for the first 2 that ^ and $ correctly match the beginning and end of a line, but they'll work on any regular expression parser, not just sed. Try them with -replace.

And I too encourage you to use Python. Getting windows scripts to run on my machine is a bit inconvenient. Last time I tried to boot into windows it was having graphics issues. Another scripting language is fine too. Even C/C++ is portable and C++11 has a graphics library.

[thistleknot]

how do I go about swapping your -e lines with the -replace function?

I don't see how the -e is specifying what to match and replace with.

As a workaround, I was going to install cygwin sed for gnuwin32

Update
okay,

sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g" *.txt does something.

I suppose one would have to do a

sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g" file.txt > file.out if wanted on a per file basis

Update
http://ask.metafilter.com/8335/Batchcorrect-files

A way to batch process and leave source files alone

[King Mir]

Ok, I now have code that will turn a folder full od modded DF installs into a much smaller folder full of mods conforming with the format at the top of the thread.  That is, 'LNP/Mods/$mod' contains only files with 'readme' or 'config' in the name, and a raw folder containing those files that are not identical to vanilla. 

For Accelerated Modest Mod (40.08), that shrinks it from 10MB to 2MB.  Not bad for such naive code!

Code: [Select]

import os
import shutil
import filecmp

# path from script to mods folder
mods_folder = 'LNP/Mods/'

mod_folders_list = []
for mod in os.listdir(mods_folder):
    if mod.startswith('df_'):
        # there must be only one folder beginning 'df_', the basis of comparison
        vanilla_folder = mods_folder + mod
        vanilla_raw_folder = vanilla_folder + '/raw/'
    else:
        mod_folders_list.append(mod)


def simplify_mod_and_df_folders():
    for mod in os.listdir(mods_folder):
        files_removed = 0
        folders_removed = 0
        for item in os.listdir(mods_folder + mod):
            # delete anything top-level not containing string 'raw', 'readme', or 'config'
            if item == 'raw':
                pass
            elif 'readme' in item.lower():
                pass
            elif 'config' in item.lower():
                pass
            elif os.path.isfile(mods_folder + mod + '/' + item):
                os.remove(mods_folder + mod + '/' + item)
                files_removed += 1
            else:
                shutil.rmtree(mods_folder + mod + '/' + item)
                folders_removed += 1
        if not files_removed + folders_removed == 0:
            print(mod, 'folder simplified!  (removed', files_removed, 'files and', folders_removed, 'folders)')

def remove_vanilla_files_from_mod_raws():
    for mod in mod_folders_list:
        mod_raw_folder = mods_folder + mod + '/raw/'
        files_removed = 0
        for file_tuple in os.walk(mod_raw_folder):
            for item in file_tuple[2]:
                item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(mod_raw_folder, '')
                if os.path.isfile(vanilla_raw_folder + item_path_str): # if the file exists in the vanilla raws
                    if filecmp.cmp(mod_raw_folder + item_path_str, vanilla_raw_folder + item_path_str): # and it's the same file
                        os.remove(mod_raw_folder + item_path_str)
                        files_removed += 1
        if files_removed > 0:
            print('\n' + mod + ':\n    removed', files_removed, 'files identical to vanilla raws')

def make_raws_with_mods():
    print('What mods do you want to load?')
    for mod in mod_folders_list:
        print('    ', mod_folders_list.index(mod), mod)
    mods_to_load = input('Enter the indicies in load order seperated by spaces.\n    ')

    # gives ordered list with indicies as strings
    mods_to_load = mods_to_load.split(' ')

    print('Just kidding, this doesn\'t do anything yet!')


simple_folders_q = input('Do you want to simplify mod folders? (y/n)\n    ')
if simple_folders_q == 'y':
    simplify_mod_and_df_folders()
    print('  Done!\n')
else:
    print('Not simplifying mod folders\n')

remove_files_q = input('Do you want to remove files from mods which are identical to vanilla raws? (y/n)\n    ')
print(remove_files_q)
if remove_files_q == 'y':
    remove_vanilla_files_from_mod_raws()
    print('  Done!\n')
else:
    print('Not removing vanilla files\n')

make_raws_with_mods()


Looks good. One small fix:
xxx.replace(mod_raw_folder, '') -> xxx[len(mod_raw_folder):]

[King Mir]

how do I go about swapping your -e lines with the -replace function?

I don't see how the -e is specifying what to match and replace with.

As a workaround, I was going to install cygwin sed for gnuwin32

-e is not a function. It's just telling sed that each of those is a separate regular expression commands to apply. The "s" however, is.

I'd expect do to it with -replace i'd look something like this:

Code: [Select]

$_ -replace  "^[^[]*",""
$_ -replace  "][^\[]*$","]"
$_ -replace  "][^[]*\[","]\n\["

Installing cygwin is great, but a bit overkill for this. It's quite a large download under default settings. (and removing everything is a pain)

[thistleknot]

I got your function to work, but by removing comments, it removes the 1st line of the file, the file_name.

I basically ran

sed -i -e "s/][^[]*\[/]\n\[/g" *.txt

I tried both scripts btw.

The original idea was to ensure that each *[ or ]*, * representing any character.  Which can be tricky due to whitespace, but generally tabs don't count, and inject a newline between the bracket and *.


To resolve it, the filename would have to be injected into the 1st line after the sed argument.

It also has the weird behaviour of putting in blank lines at the top of the outputted files.

Update
Okay... it seems to have worked with one file when it just outputs to the stream, but still has the issue of not properly putting comments on their own line

Code: [Select]

item_gloves

[OBJECT:ITEM]

###test###[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
[NAME:gauntlet:gauntlets]
[ARMORLEVEL:2]
[UPSTEP:1]
[SHAPED]
[LAYER:ARMOR]
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
I ran

sed -e "s/][^[]*\[/]\n\[/g" *.txt