Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[King Mir]

Here's a sed (*nix utility) command that will do it:

Code: [Select]

sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"It removes comments and splits tokens to one per line. To just split up tokens you can do this:

Code: [Select]

sed -e "s/][^[]*\[/]\n\[/g"
You can also use those regular expressions with any regular expression library.

EDIT: But instead of doing this all in shell scripts, it would be better and more portable to use Python.

[Merkator]

I was thinking about some more complex version of extended diff format.
For example:

Code: [Select]

raw>object>entity_default.txt
?[ENTITY:MOUNTAIN]
+[NOPAIN]
-[PERMITTED_REACTION:SMELT_IRON] # just sample.
@[ENTITY:DWARVES]

Where

Code: [Select]

     ? find line
     + mean insert line
     -  remove line
     @ replace line
     # comment

This can be easily connected with each other.
For example if you want replace tag ITEM_WEAPON:Sword with ITEM_WEAPON:Big Sword
You need to just write it like that

Code: [Select]

raw>object>item_weapon.txt
?[ITEM_WEAPON:Sword]
@[ITEM_WEAPON:Big Sword]

It should be much easier that regexp.

[thistleknot]

I was thinking about some more complex version of extended diff format.
For example:

Code: [Select]

raw>object>entity_default.txt
?[ENTITY:MOUNTAIN]
+[NOPAIN]
-[PERMITTED_REACTION:SMELT_IRON] # just sample.
@[ENTITY:DWARVES]

Where

Code: [Select]

     ? find line
     + mean insert line
     -  remove line
     @ replace line
     # comment

This can be easily connected with each other.
For example if you want replace tag ITEM_WEAPON:Sword with ITEM_WEAPON:Big Sword
You need to just write it like that

Code: [Select]

raw>object>item_weapon.txt
?[ITEM_WEAPON:Sword]
@[ITEM_WEAPON:Big Sword]

It should be much easier that regexp.

while I agree with you logging the token id with the patch file.  Peredexis was concerned with the drama involved with modifying the patch format.

I believe by flattening the raw file, it allows for contextual block matching to take over.  So, in theory, creatures should line up and token differences should be noticed more.  It might not really be the ideal solution and may require additional parsing (such as token comparisons between the same object between two mods).

[thistleknot]

Here's a sed (*nix utility) command that will do it:

Code: [Select]

sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"It removes comments and splits tokens to one per line. To just split up tokens you can do this:

Code: [Select]

sed -e "s/][^[]*\[/]\n\[/g"
You can also use those regular expressions with any regular expression library.

EDIT: But instead of doing this all in shell scripts, it would be better and more portable to use Python.

thx, linux cl works in powershell apparently
nm

[thistleknot]

okay, final udpate.

I worked really hard to ensure when I split token's that it didn't create [empty] newlines.

basically, tabs are removed first, then a carriage return "`r" and new line "`n" is created inbetween ]['s

works in powershell

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
        $out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"

        Get-Content $file | Foreach-object {
            $_ -replace "`t","" `
-replace '\]\[',"]`r`n["
        } | Set-Content $outFile
    }




output is .txt2

sample before and after output

Spoiler: before (click to show/hide)

Code: [Select]

reaction_other

[OBJECT:REACTION]

[REACTION:TAN_A_HIDE]
[NAME:tan a hide]
[BUILDING:TANNER:CUSTOM_T]
[REAGENT:A:1:NONE:NONE:NONE:NONE][USE_BODY_COMPONENT][UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:TAN_MAT]

In the product, if you want to use the reagent's material itself, use NONE instead of a reaction product class (TAN_MAT in this example).

[PRODUCT:100:1:SKIN_TANNED:NONE:GET_MATERIAL_FROM_REAGENT:A:TAN_MAT]
[SKILL:TANNER]
[AUTOMATIC]

[REACTION:RENDER_FAT]
[NAME:render fat]
[BUILDING:KITCHEN:CUSTOM_R]
[REAGENT:A:1:GLOB:NONE:NONE:NONE][REACTION_CLASS:FAT][UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:RENDER_MAT]
[PRODUCT:100:1:GLOB:NONE:GET_MATERIAL_FROM_REAGENT:A:RENDER_MAT]
[SKILL:COOK]
[AUTOMATIC]

[REACTION:MAKE_SOAP_FROM_TALLOW]
[NAME:make soap from tallow]
[BUILDING:SOAP_MAKER:CUSTOM_T]
[REAGENT:lye:150:LIQUID_MISC:NONE:LYE]
[REAGENT:lye container:1:NONE:NONE:NONE:NONE]
[CONTAINS:lye]
[PRESERVE_REAGENT]
[DOES_NOT_DETERMINE_PRODUCT_AMOUNT]
[REAGENT:tallow:1:GLOB:NONE:NONE:NONE][REACTION_CLASS:TALLOW][UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:SOAP_MAT]
[PRODUCT:100:1:BAR:NONE:GET_MATERIAL_FROM_REAGENT:tallow:SOAP_MAT]
[PRODUCT_DIMENSION:150]
[SKILL:SOAP_MAKING]


Spoiler: after (click to show/hide)

Code: [Select]

reaction_other

[OBJECT:REACTION]

[REACTION:TAN_A_HIDE]
[NAME:tan a hide]
[BUILDING:TANNER:CUSTOM_T]
[REAGENT:A:1:NONE:NONE:NONE:NONE]
[USE_BODY_COMPONENT]
[UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:TAN_MAT]

In the product, if you want to use the reagent's material itself, use NONE instead of a reaction product class (TAN_MAT in this example).

[PRODUCT:100:1:SKIN_TANNED:NONE:GET_MATERIAL_FROM_REAGENT:A:TAN_MAT]
[SKILL:TANNER]
[AUTOMATIC]

[REACTION:RENDER_FAT]
[NAME:render fat]
[BUILDING:KITCHEN:CUSTOM_R]
[REAGENT:A:1:GLOB:NONE:NONE:NONE]
[REACTION_CLASS:FAT]
[UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:RENDER_MAT]
[PRODUCT:100:1:GLOB:NONE:GET_MATERIAL_FROM_REAGENT:A:RENDER_MAT]
[SKILL:COOK]
[AUTOMATIC]



Update
I did some tests with trying a 3 way merge.  The merging process is a lot smoother with the flattened raws, but I could still see a need for still checking if the token's in conflicts are being duplicated.  That would resolve most conflicts I think.

Update 2
I took a walk and thought about why I was having [merge] conflicts.  I was wondering if, using this parse method, it was due to not setting all tokens on their own line by themself when they were next to a commented out line... 

however, the merge conflicts I had were not due to token's being next to a comment (I think I might have to update the parse method to consider if a token is adjacent to a comment).

I think it might be because I applied two mods (patch of 34_11 to accelerated, and a patch of accelerated to modest...as one patch over a 3rd mod (40_08)), which I think may have cause some [merge] conflicts with [duplicate] grazer tags.

In other words, I'm thinking if raws are parsed using this method, and applied one at a time to a 3rd mod.

For ex.

diff of 34_11 vs 34_11 accelerated: as a patch
  applied to 40_08
diff 34_11 accelerated vs 34_11 accelerated modest: as a patch
  applied to 40_08 accelerated

one might avoid conflicts?  I certainly believe raw parsing the files in this manner before merging reduces the # of contextual conflicts.

Here's what a diff between 34_11 and 34_11 accelerate modest looks like with this parse.

http://pastebin.com/curfNcMH

Update 3:
I think if one back-update's some in house object tracking variables (using these patch files); such as file_name, and the first [object: name] token, using a scripting language like python, one should be able to fill in the blanks when it comes to patching location (even without col, row information)

Then one could parse the diff files and derive the object id, and be able to ensure the diff file is being applied to the correct object/token id (such as: creature or item or entity, or whatever) by keeping track of duplicate[ token]s!

That would resolve the grazer issue.  If castes come into play...  I guess a simple check if the current caste is having a token duplicated, then figure out what token is the master token to be used.

[PeridexisErrant]

The general impression I'm getting is that we can get quite a long way with standard libraries, and don't need to write complex token-based logic at this stage. 

Flattening clearly makes a standard diff much more powerful.  It's unclear to me how the mixed raws might later by un-flattened, and whether this is important.  The cases where flattening means more context is required are somewhat concerning, but hopefully not a show stopper.  I suggest we basically set the flattening aside for now, since it can be added back into the middle of the logic later without too much trouble and having a working codebase to add it to will help. 

I propose, for version 2.0 - ie NOT YET -
 - adding to the standard the *exact* regexp used to flatten the raws, (once it's agreed after testing)
 - that flattened raws go in the mod's folder in "/raws_flat/",
 - once we have an exact diff that also becomes part of the standard
 - diff-only mods go in "/raws-diffs/"
 - packs can then include any or all of these
 - the launcher must be able to accept any of these formats
 - but may use any of those present (since it must make no difference to the outcome)

I think it's worth distinguishing between a no-conflict n-way merge, and sequential (overwriting) simple merges.  The GUI should highlight the former green, the latter orange, and impossible merges red; ideally this is also applied to subsets (eg mods 1-3 green, 4 and five yellow, six red) to give an indication of compatibility.  A stretch goal might be to find the least-conflicting load order for the user.  Part of the goal of this system is to make mods easy, and this simple feedback would help a lot. 

If at all possible, it would be good to post code snippets in Python - shell scripts or whatever are fine for testing, but it's all got to be Python eventually and in many cases you can just import $tool and avoid an ugly workaround. 

[hermes]

Just want to chime in with a hope for this great project...  Please make the process of integrating/formatting mods into this system very very easy.  I feel that over the years the DFHack and Masterwork projects have basically turned modding into programming, taking away what made it accessible and fun in the beginning.  It's be a shame if there was yet another barrier to entry in the form of a standard which everyone had to learn and comply to. 

[PeridexisErrant]

Please make the process of integrating/formatting mods into this system very very easy.

It'd be a shame if there was yet another barrier to entry in the form of a standard which everyone had to learn and comply to.

This is a core part of the project:  the format is simply a normal raw folder, containing only changed files.  The launcher should be able to deal with a full install as well, by simply deleting the other files.  This makes creating and adding mods very very easy.  The standard should be trivial to comply with, since it's only what a mod needs anyway, and all the potentially-difficult manipulation is done in the program. 

[PeridexisErrant]

Hi all.  First code - this simplifies the mod folders and df folders down to a raw folder, but leaves top-level files whose names include 'readme' or 'config'.  Oh, and builds a list of mods and vanilla raws, but that doesn't come into this bit. 

Next plan is to remove any files in any raw folder under the mods list which exactly matches the corresponding vanilla file, after checking that there is only one vanilla folder to compare to!

Code: [Select]

import os
import fnmatch
import fileinput
import shutil

mod_folder = 'LNP/Mods/'

mod_folders_list = []
vanilla_folders_list = []
for mod in os.listdir(mod_folder):
    if mod.startswith('df_'):
        vanilla_folders_list.append(mod)
    else:
        mod_folders_list.append(mod)


def simplify_mod_and_df_folders():
    for mod in os.listdir(mod_folder):
        files_removed = 0
        folders_removed = 0
        for item in os.listdir(mod_folder + mod):
            # delete anything top-level not containing string 'raw', 'readme', or 'config'
            if 'raw' in item:
                pass
            elif 'readme' in item.lower():
                pass
            elif 'config' in item.lower():
                pass
            elif os.path.isfile(mod_folder + mod + '/' + item):
                os.remove(mod_folder + mod + '/' + item)
                files_removed += 1
            else:
                shutil.rmtree(mod_folder + mod + '/' + item)
                folders_removed += 1
        if files_removed + folders_removed == 0:
            print(mod, 'folder is already simplified')
        else:
            print(mod, 'folder simplified!  (removed', files_removed, 'files and', folders_removed, 'folders)')

simplify_mod_and_df_folders()


[King Mir]

I don't think you want to modify the mods. Just ignore the parts that you don't care about. So you want to create a list of valid files files that are to be merged, but you don't want to remove everything else.

[PeridexisErrant]

I don't think you want to modify the mods. Just ignore the parts that you don't care about. So you want to create a list of valid files files that are to be merged, but you don't want to remove everything else.

The goal is to imitate the LNP graphics loading; it works with a full install being referenced but you can choose to simplify / delete inactive stuff if you want to to save space.  Later the removal functions can be called on temporary copies for processing if the user doesn't want to delete this stuff, but for now I'm building them and don't mind testing directly. 

Next up is to detect and remove vanilla files (likewise movable to temp files later), and then the first merge logic: overwrite a copy of the vanilla folder with the mod.  Actual merging comes later

[King Mir]

So the psudocode I imagine is this:

Code: [Select]

string vanilla_raw_location //constant
string generated_raw_location //constant
mod_folders_list = find_mod_folders()
mod_raw_paths = get_raw_paths(mod_folders_list)

copy_vanilla_raws(vanilla_raw_location, generated_raw_location)

for each mod_raw_path in mod_raw_paths
  for each raw_relative_path in get_valid_raw_files(mod_raw_path)

      from_file = mod_raw_path  + raw_relative_path
      to_file = generated_raw_location + raw_relative_path
      vanilla_file = vanilla_raw_location + raw_relative_path

      if file(to_file) does not exist
         copy_file(from_file, to_file)
      else if(file(vanilla_file) does not exist
         cleanup_and_abort()
      else if(diff3 (from_file,vanilla_file,to_file) does not merge smoothly
         cleanup_and_abort()
      else
         merge(from_file,to_file)         


[King Mir]

I don't think you want to modify the mods. Just ignore the parts that you don't care about. So you want to create a list of valid files files that are to be merged, but you don't want to remove everything else.

The goal is to imitate the LNP graphics loading; it works with a full install being referenced but you can choose to simplify / delete inactive stuff if you want to to save space.  Later the removal functions can be called on temporary copies for processing if the user doesn't want to delete this stuff, but for now I'm building them and don't mind testing directly. 

Next up is to detect and remove vanilla files (likewise movable to temp files later), and then the first merge logic: overwrite a copy of the vanilla folder with the mod.  Actual merging comes later

You're just doing extra work that way. Trimming functions may be useful for somethings, but you don't need them here. You just need a list of valid files specified as a mod-relative path.

[PeridexisErrant]

You're just doing extra work that way. You just need a list of valid files specified as a mod-relative path.

Quite possibly.  However I'd rather have working code to refactor than spend years in design, so...

[thistleknot]

even better script guys.

Thanks to briantist on stackexchange

http://stackoverflow.com/questions/25345739/powershell-regexp-to-replace-any-character-left-bracket-and-replace-wit

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
        $out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"

        Get-Content $file | Foreach-object {
            $_ -replace '(?m)^\s*','' `
-replace '(\[.+?\][^\[\r\n]*)(?=\[)' , "`$1`r`n"
        } | Set-Content $outFile
    }
   

Spoiler: what it does (click to show/hide)

   

Code: [Select]

$inputText = @"
reaction_other

[OBJECT:REACTION]

[REACTION:TAN_A_HIDE]
    [NAME:tan a hide]
    [BUILDING:TANNER:CUSTOM_T] ## Some comment here
    [REAGENT:A:1:NONE:NONE:NONE:NONE] # Here [USE_BODY_COMPONENT] ## Inline comment [UNROTTEN]
        [HAS_MATERIAL_REACTION_PRODUCT:TAN_MAT]
"@

$outputText = $inputText -replace '(?m)^\s*','' `
              -replace '(\[.+?\][^\[\r\n]*)(?=\[)' , "`$1`r`n"
How This Works

It's still being done in 2 -replaces.
Replace 1:

The first -replace removes all of the leading whitespace:

    (?m) is the inline regex mode specifier where m stands for "multiline" and it effectively makes ^ and $ match the beginning and end of each line rather than the beginning and end of the whole string.
    ^ is therefore matching the beginning of each line.
    \s is a character class that stands for most whitespace, including spaces and tabs.
    * means zero or more.

So effectively this says:

    Replace all whitespace at the beginning of each line with an empty string (effectively removing it).

Replace 2:

    We start by opening a capturing group with (.
    Next we match a literal left bracket with \[, followed by 1 or more of any character (non-greedy), followed by a literal right bracket \].
    Still in our capturing group, we start a character class by using [ (not escaped), and in this context ^ means NOT, so whatever we put inside the class must not be present.
    Inside the character class, we use a literal left bracket \[, a CR \r and a LF \n, then close the class ]. We use * to indicate that we want 0 or more of that class.
    The capturing group is now closed.
    This next part is a positive lookahead. It looks to see if there is a particular match coming up, but the critical part here is that it does not consume those characters, therefore they are not part of the match, therefore they will not be replaced.
    The replacement string is the value of the first capture group, followed by CRLF. Note the double quoted string. This requires us to escape the $ using backtick, so that powershell doesn't interpret $1 as a powershell variable. Instead a literal $1 string will be sent to the regex engine, where it will interpret as a backreference.

So this replace is basically saying:

    Find a string inside of square brackets

• optionally followed by any amount of text that is not a [ or a line ending. But, only count it as a match if there is a [ directly following the matched string. If all conditions are met, replace that string with itself followed by CRLF.

input: *.txt

Spoiler (click to show/hide)

reaction_other

[OBJECT:REACTION]

[REACTION:TAN_A_HIDE]
   [NAME:tan a hide]
   [BUILDING:TANNER:CUSTOM_T]
   [REAGENT:A:1:NONE:NONE:NONE:NONE][USE_BODY_COMPONENT][UNROTTEN]
      [HAS_MATERIAL_REACTION_PRODUCT:TAN_MAT]

   In the product, if you want to use the reagent's material itself, use NONE instead of a reaction product class (TAN_MAT in this example).

   [PRODUCT:100:1:SKIN_TANNED:NONE:GET_MATERIAL_FROM_REAGENT:A:TAN_MAT]
   [SKILL:TANNER]
   [AUTOMATIC]

[REACTION:RENDER_FAT]
   [NAME:render fat]
   [BUILDING:KITCHEN:CUSTOM_R]
   [REAGENT:A:1:GLOB:NONE:NONE:NONE][REACTION_CLASS:FAT][UNROTTEN]
      [HAS_MATERIAL_REACTION_PRODUCT:RENDER_MAT]
   [PRODUCT:100:1:GLOB:NONE:GET_MATERIAL_FROM_REAGENT:A:RENDER_MAT]
   [SKILL:COOK]
   [AUTOMATIC]

output: *.txt2

Spoiler (click to show/hide)

reaction_other

[OBJECT:REACTION]

[REACTION:TAN_A_HIDE]
[NAME:tan a hide]
[BUILDING:TANNER:CUSTOM_T]
[REAGENT:A:1:NONE:NONE:NONE:NONE]
[USE_BODY_COMPONENT]
[UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:TAN_MAT]

In the product, if you want to use the reagent's material itself, use NONE instead of a reaction product class (TAN_MAT in this example).

[PRODUCT:100:1:SKIN_TANNED:NONE:GET_MATERIAL_FROM_REAGENT:A:TAN_MAT]
[SKILL:TANNER]
[AUTOMATIC]

[REACTION:RENDER_FAT]
[NAME:render fat]
[BUILDING:KITCHEN:CUSTOM_R]
[REAGENT:A:1:GLOB:NONE:NONE:NONE]
[REACTION_CLASS:FAT]
[UNROTTEN]
[HAS_MATERIAL_REACTION_PRODUCT:RENDER_MAT]
[PRODUCT:100:1:GLOB:NONE:GET_MATERIAL_FROM_REAGENT:A:RENDER_MAT]
[SKILL:COOK]
[AUTOMATIC]

this method should hopefully address comments onto their own lines as well.

Update

so... to test these changes out yourself, install a git client (I use github, it's all I know).

and you can derive your own patch files after parsing your raws and replacing the old ones with generated ones

how?
install a git client
create a commit based on vanilla raws
run the above script on your copy of [vanilla] raws
replace yours raws with the "flattened" raws
recommit
Create a branch based on your old client (can be done using the filter menu by typing in a new branch if using github)
open a git console, and run this to derive a patch file

git diff a e > mynewpatch.patch

a being the hex commit # of pre changes
and
b being the hex commit # of post changes

ex.

a = vanilla hex commit
b = civforge hex commit

that should give you a contextual diff file

then you can use tortoisegitmerge to merge the diff file to a mod folder.

I would imagine that if there are no conflicts, that you could use diff command line to do all the merging

Update 2
I just made some merges using this parsing method, and it still requires manual intervention.  Maybe about 5 minutes worth of work to apply a mod, then derive a patch.

With this system.  Whoever implements the mod options, has to do all the manual merge conflicts themself, then derive a patch that will be ran over vanilla.

Which means version tracking for the mod creator to fine tweak the commits/new branches and base patch files off those branches.

Here's my patch files:

http://dffd.wimbli.com/file.php?id=9418

...

In all honesty, it would almost be better to use a github repository to host all the mod options from...


Otherwise, you have to basically ship out a git offline client to the end user to be able to fire up and have all the local repositories available.

Or... one could make an elaborate setup of patch files to apply, but ultimately, someone somewhere has to keep track of it using a git version tracking.

Update again
Reuploaded a new set of patches, missed the 40_08 patch accelerated (appliied)...

http://dffd.wimbli.com/file.php?id=9418