Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[King Mir]

We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install.

I do. I want to be able to easily add and remove mini-mods without having to manually reapply each mod to vanilla on each remove or to work out how to remove a mini-mod while leaving others intact.

[Pidgeot]

The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.

I think some kind of custom format is unavoidable; it just needs to be open.

We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install.  We also don't get much bang for the buck if we don't store things as diffs of some type (a graphics pack can make tiny changes to lots of large files).

It's unavoidable in the long-term, definitely, but for the short-term, Peridexis is proposing to use a completely standard diff (with the pitfalls that brings along), to provide a more easily attainable starting point.

If you're making a custom format, then you'd prefer to make sure it is as complete (or extensible) as we would ever need, and that's when it takes more time - and you would likely end up with something like (but not necessarily identical to) ModBase.

[Dirst]

King Muir, I was being a little too flippant... forgetting about the removal part.  That said, if you assume away file collisions then it could be a simple script.  (For the ultimate in simplicity, back in VMS you could accomplish this kind of rollback with a single command.)

Pidgeot, something like ModBase would be a bit too raw-aware.  I like that idea of being able to do math in the script, and I noted above that positioning should be done at the tag level, but otherwise it should be processing more-or-less arbitrary text.  Otherwise we dive down a rabbit-hole of trying to parse increasingly complex tag structures with a never-ending parade of special cases.  A couple *nix standards, slightly buffed, gets you just about everything that modders do except re-ordering things.

But I do agree, the simple diffs that can't handle collisions is a good starting point.

[thistleknot]

I did all that... with simple strings... I could reference certain values of certain tags of certain objects, like creating a sword with a damage value dependent on that of an existing sword.

[ITEM_WEAPON:ITEM_WEAPON_SSWORD_CRUEL]
[NAME:shortsword:shortswords]
[ADJECTIVE:Cruel]
[DAMAGE@:%ITEM_WEAPON_SWORD_SHORT+10:GORE]
[WEIGHT:35]
[SKILL:SWORD]
[CRIT_BOOST:1]
[TWO_HANDED:4]
[MINIMUM_SIZE:4]
[MATERIAL_SIZE:6]
[STICK_CHANCE:30]

It's context-sensitive in this case (pulling the damage value from the same spot of the same tag of a different item of the same type), but was intended to work just as well for different-type objects and different tags.

I think your tool needs to be updated to include castes then. However, I think a simple collection of "patches" that players could load on vanilla would be the next step. I do know ur tool required vanilla as a base but the steps seedseem a bit cumbersome. I would think a simple batch file should all that would be needed to apply some patches to vanilla

[milo christiansen]

One word: Rubble.

The only real lack is a cross platform GUI, but that is just the front end, the actual tool works on all OS's.

[PeridexisErrant]

The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.

I think some kind of custom format is unavoidable; it just needs to be open.

We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install.  We also don't get much bang for the buck if we don't store things as diffs of some type (a graphics pack can make tiny changes to lots of large files).

I do. I want to be able to easily add and remove mini-mods without having to manually reapply each mod to vanilla on each remove or to work out how to remove a mini-mod while leaving others intact.

It's unavoidable in the long-term, definitely, but for the short-term, Peridexis is proposing to use a completely standard diff (with the pitfalls that brings along), to provide a more easily attainable starting point.

If you're making a custom format, then you'd prefer to make sure it is as complete (or extensible) as we would ever need, and that's when it takes more time - and you would likely end up with something like (but not necessarily identical to) ModBase.

One word: Rubble. 

Custom formats are a lot of work, have been done, and have been done well.  ModBase is great, Rubble is simply brilliant, but they're solving a different problem.  The goal here is *not* to merge overlapping mods well, it's to make using mods easy for new players. 

Maybe a custom format is unavoidable in the long term, I suspect not, but until something has wide adoption any new format will lead to nothing but fragmentation and stagnation as it becomes too much trouble to use these tools.  In any event, if a special format was used at all I would need to hear some pretty good reasons to do anything but include Rubble and give users the option of the simple and advanced mod loaders.  In response to all the ideas about flattening raw structures, maths in scripts, etc:  different goals.  The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging.  Making two unrelated changes to a file is enough!

The advantage of a diff over simple file overwrites is simply that it handles bloated small mods more elegantly, and it's fairly easy to implement with standard libraries.  If not, file overwrites will do.  I'm hoping we'll also see a slightly different mod philosophy where instead of including all the favourite tweaks, we get just the changes that are a core part of the mod, equivalent to graphics packs leaving the population cap at vanilla levels. 

Other scripts, fancy tools, etc should be just as compatible with the output of this process as any other mod; so scripts which (eg) edit aquifer tags will work just fine. 

[milo christiansen]

I still think that Rubble is just a good GUI away from solving the problem, and for those who are stuck on diffs Rubble can (as of 4.4) apply standard format patches.

It is easy to look at Rubble as a modding tool, but I designed it as a general purpose mod installer first and a modding tool second. The only drawback is that mods need to be prepared to work with it, you cannot just drop any old mod in and expect it to work.

If someone comes up with a general purpose mini-mod format that does not depend on a specific scripting system or other host I would be happy to add Rubble support for it, just post in the Rubble thread when finalized.

[King Mir]

In response to all the ideas about flattening raw structures, maths in scripts, etc:  different goals.  The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging.  Making two unrelated changes to a file is enough!

I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.

[PeridexisErrant]

In response to all the ideas about flattening raw structures, maths in scripts, etc:  different goals.  The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging.  Making two unrelated changes to a file is enough!

I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.

For clarity, I'm opposed to requiring this in the input format but think it's a good idea in the logic. 

It should be easy enough to flatten all the raws in memory or create temporary copies processed in whatever way we like, which are then handled however.  "merge(flatten(mod1), flatten(mod2))" rather than "merge(flat_mod1, flat_mod2)".  Same effect, easy enough to code, more consistent, easier and more flexible for modders. 

[thistleknot]

I'm thinking instead... of xml files.

The raws should be loaded into xml's, comments and all.  The goal is to preserve original file structure (comments and their contextual positions) and token information.

Then some custom patch files are based on the differences based on the xml token's.

Then you have a standardized format that is based on something that works with xml encapsulation vs some other arbitrary patch algorithm.  Only catch is, someone would have to make xml structure to read the raw files (I'm reading up on TinyXML, but not necessarily volunteering). 

Then another program to measure differences between the two xml's of each compared file.

Of course then another overall utility to implement those changes in a way similar to rubble, mod manager, or mod base.

[PeridexisErrant]

So long as there's a rigidly defined way to translate raws into xml format, I guess that could work well.  I'm still not touching that kind of thing until after we have a mod loader 1.0 out and working. 

[thistleknot]

look at this from some rawexplorer link I was viewing

http://dftokens.gumpstudio.com:8080/ViewToken.aspx?Token=PHILOSOPHER

[King Mir]

In response to all the ideas about flattening raw structures, maths in scripts, etc:  different goals.  The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging.  Making two unrelated changes to a file is enough!

I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.

For clarity, I'm opposed to requiring this in the input format but think it's a good idea in the logic. 

It should be easy enough to flatten all the raws in memory or create temporary copies processed in whatever way we like, which are then handled however.  "merge(flatten(mod1), flatten(mod2))" rather than "merge(flat_mod1, flat_mod2)".  Same effect, easy enough to code, more consistent, easier and more flexible for modders.

Yeah that's what I meant. Flatten the mods in the tool. 

I'm mostly sold on this proposal now. My biggest concern is if it'll be good enough to handle graphic packs, but worst case you could use that as the baseline instead of vanilla. Maybe I'll have other concerns after I sleep on it more.

[thistleknot]

I did some searching for a powershell script that could be modified to accomplish that for a set of raws.

http://stackoverflow.com/questions/15283931/powershell-find-replace-string-with-carriage-return-and-line-feed

Spoiler (click to show/hide)

Code: [Select]

$filenames = @("E:\blergfest.csv")
foreach ($file in $filenames) {
    $outfile = "$file" + ".out"

    Get-Content $file | Foreach-object {
        $_ -replace '\~}}|{{E', "`r`nE" `
            -replace '\|\r\n', "`r`n"
    } | Set-Content $outfile
}
I'll update the post with a working version if I get it working.

Update
This works

and you can modify it to queue up all the files you need!

Code: [Select]

$filenames = @("creature_standard.txt")
    foreach ($file in $filenames) {
        $outfile = "$file" + ".out"

        Get-Content $file | Foreach-object {
            $_ -replace '\[',"`r`n" `
-replace '\]',"`r`n"
        } | Set-Content $outfile
    }


Spoiler (click to show/hide)

this version leaves brackets in, but I can only guess the large spaces between the brackets [of the output] represents whitespace.

Code: [Select]

$filenames = @("creature_standard.txt")
    foreach ($file in $filenames) {
        $outfile = "$file" + ".out"

        Get-Content $file | Foreach-object {
            $_ -replace '\[',"[`r`n" `
-replace '\]',"`r`n]"
        } | Set-Content $outfile
    }

if your feeling particularly dangerous, you can modify this portion below to have it search all files/folders beneath subdirectory.

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files)


Okay, here's the powershell script that will process your entire subdirectory and flatten all the files as you described.

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
        $outfile = "$file" + ".out"

        Get-Content $file | Foreach-object {
            $_ -replace '\[',"[`r`n" `
-replace '\]',"`r`n]"
        } | Set-Content $outfile
    }
You know what's funny, is how would one reverse that?  Say you didn't want to leave the raws looking like that after you merged them.  How would one reverse this formula?  I can't merely swap the  '\['   with   "[`r`n"

Update

by doing this [conversion] to new lines, contextual differences might have to be extended to 4 or 5 or maybe even more lines when deriving diff's.

Carrying further on this idea.

http://www.thinkplexx.com/learn/howto/scm/svn/how-to-create-and-use-local-svn-subversion-repository-on-windows-or-linux-simple-and-fast-step-by-step

One could use a local svn repository system to apply these mods.  Possibly even create little git batch scripts with a front end menu to load configurations of mods using a cherry picking system.

Update
I tried it out on github to see what a 34.11 to 40_08 accelerated modest mod conversion would look like and it has less conflicts, but, breaking up the token's has chaotic affects, especially with conflicting tokens.  Two token's are kind of wanted to be merged together.

http://imgur.com/yLFixQO

It would also tremendously help by first parsing all tokens from non tokens, and just working with tokens so as to avoid any merging errors with comments and flattened tokens.

[thistleknot]

I fixed that script a little more (was putting the [ and ]'sin the wrong place)...

Code: [Select]

$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
        $outfile = "$file" + ".out"

        Get-Content $file | Foreach-object {
            $_ -replace '\[',"`r`n[" `
-replace '\]',"]`r`n"
        } | Set-Content $outfile
    }
sample output

Code: [Select]

c_variation_default


[OBJECT:CREATURE_VARIATION]



[CREATURE_VARIATION:ANIMAL_PERSON]


[CV_REMOVE_TAG:NAME]


[CV_REMOVE_TAG:GENERAL_CHILD_NAME]


[CV_REMOVE_TAG:GENERAL_BABY_NAME]


[CV_REMOVE_TAG:CASTE_NAME]


[CV_REMOVE_TAG:CHILDNAME]


[CV_REMOVE_TAG:BABYNAME]


[CV_REMOVE_TAG:SMALL_REMAINS]


[CV_REMOVE_TAG:DESCRIPTION]


[CV_REMOVE_TAG:CREATURE_TILE]


[CV_REMOVE_TAG:COLOR]


[CV_REMOVE_TAG:MAXAGE]


[CV_REMOVE_TAG:SOUND]

I'm thinking I can modify this to remove all blank lines

Spoiler: update on blank line removal (click to show/hide)

using a batch file, I was able to focus on strings in a .txt file, but... whitespace is being counted as strings as well

test.bat

Code: [Select]

for %%f in (*.out) do (
findstr "." %%~nf.txt2 > %%~nf.out2)


I may... Don't hold me to it, be able to indent object type using a regular expression

Then use a batch file to derive diff's between two folders to generate a unified diff patch