Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[thistleknot]

Here is the sed answer:

updated at 12:02PM PST

Code: [Select]

sed -e "s/\t//g" -e "s/(?m)^\s*//g" -e "s/\]\(.\)/]\r\1/g;s/ *\[/[/g;s/\(.\)\[/\1\r[/g" %%~na.txt > %%~na.out
requires gnuwin32 sed

http://http://sourceforge.net/projects/gnuwin32/files//sed/4.2.1/sed-4.2.1-setup.exe/download

Code: (ParseRaws.bat) [Select]

echo off
for /f %%a in ('dir /b *.txt') do sed -e "s/\t//g" -e "s/(?m)^\s*//g" -e "s/\]\(.\)/]\r\1/g;s/ *\[/[/g;s/\(.\)\[/\1\r[/g" %%~na.txt > %%~na.out
erase *.txt
ren *.out *.txt
REM remove all blanklines
REM -e  "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d"
echo on

doesn't parse subfolders due to the way it grabs a list of files...

before input:

Code: [Select]

item_gloves

[OBJECT:ITEM]

###test###[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]###test###
[ARMORLEVEL:2][UPSTEP:1]###test###
[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]

after output:

Code: [Select]

item_gloves

[OBJECT:ITEM]

[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]

Spoiler: On an important note, removing 'comments' IS BAD, look at book_art.txt (click to show/hide)

Code: [Select]

[NAME]
[PHRASE]
[ADJ]
[NO_ART_NAME]
The [ADJ]
[NO_ART_NAME]
[NAME]
[ADJ]
[NAME] and the
[NOUN]
[NAME] and the
[ADJ]
[NOUN]
The [NOUN] and
[NAME]
The [ADJ]
[NOUN] and
[NAME]
[NAME]:
[PHRASE]
It Must Have Been [NAME]
My Friend [NAME]
The Birth of [NAME]
The Sun Sets on [NAME]
We See [NAME]
[NAME] Ever Onward
[NAME] and Nothing More
And [ANY_PRONOUN] Sang '
[NAME]!'



[Dirst]

Wow, I step away for an evening and come back to four pages of unread posts.  Impressive work so far, it looks like flattening has been solved.  We don't need an omnipotent solution that's going to mix Masterwork, OldGenesis and combat balancing from 23a into a single error-free 40.08 game... but a little simple merging would go a long way to making mini-mods play nice with each other.

I thought of something while I didn't have access to my machine.  There are two reasons why N-way merged files break.

Case 1: Mod A adds or deletes lines before mod B tries to change something.  Mod B's changes end up in the wrong place, which may or may not cause issues.

Case 2: Mod A and mod B try to change the same line.

Of course, both could happen in the same merge.  In this case mod B's changes happen at the wrong line and we could potentially end up with a duplicated raws problem.

I have a gut feeling that someone has already tackled case 1 in a standard library somewhere.  My initial guess of a solution would be to do everything in two passes.  Clone the original file to an intermediate file, tag the intermediate file with where each change is supposed to occur, then apply those changes whereever the tags ended up, stripping off the tags as we go.  If two mods try to change the same line, last in wins (and this can be logged for the player).

So, a snippet of the tagged intermediate file might look like

Code: [Select]

{RampageMod:13}[LARGE_ROAMING]
[BIOME:ANY_TROPICAL_FOREST]
{Zootastic:2}[BIOME:SHRUBLAND_TROPICAL]
{Zootastic:3}{RampageMod:14}[POPULATION_NUMBER:15:30]
{Zootastic:4}{RampageMod:15}[CLUSTER_NUMBER:3:7]
{RampageMod:16}[BENIGN]
[MEANDERER]
[NATURAL]
{Zootastic:5}[PREFSTRING:strength]

This should ensure that changes land where the modder thinks they will based on a vanilla diff.  Won't ensure that the changes are compatible, but at least they won't be off in some other object.

Does that make sense to anyone other than me?

[thistleknot]

I'm working on this issue right now.

I talked about if we merge patch files before applying them [for n-way merges], we address the conflicts at the patch level.

there's a linux tool called combinediff
http://cyberelk.net/tim/patchutils/man/rn01re02.html

I'm wondering if there's a way to do this with github...

anyways, my idea was:

34_11 to accelerated = diff1
34_11 to 40_08 = diff2

combinediff diff1 and diff2 to diff3

apply diff3 to 34_11
have 40_08 accelerated

why/who would do this?

Someone who wants a 40_08 accelerated to derive a patch based on 40_08.  The end user wouldn't need the 34_11 raws btw.  This would be done from the patch distributor's end

[King Mir]

Case 1 is a special case of case 2.

The plan to handle case 2 is a run a 3 way diff and reject the file if there are any overlaps. In my psudocode I assumed that it's sufficient to compare the mod being added to the generated raws with vanilla as a baseline. So there's no intermediate file, except the output, but you do keep a copy of vanilla to compare with. A narrow test for overlapping would be to make sure the generated raws and mod raws of a new mod being added don't have consecutive changes, that is they don't both modify vanilla at the same place.

It occurs to me that a raw aware check for duplicate raws may be unavoidable for correctness. Otherwise you could have mods that try to add the same thing, in different files, and no differ is doing to realize it. Especially if they are not identical. Of course one option is to ignore this case.

[thistleknot]

I do not think combinediff is exactly what I thought it was I think instead it just merges two sequential merges into one merge.

So no silver bullet on that.

So... no "ala cart merge of mods, if those mods have conflicts" and being able to add on any combination of things you want.

Instead, I would recommend the mod author or tool author, take a poll and ask the community what base of mods should be offered to the community, because most likely, it will be a set # of options & combinations.

[Dirst]

I do not think combinediff is exactly what I thought it was I think instead it just merges two sequential merges into one merge.

So no silver bullet on that.

So... no "ala cart merge of mods, if those mods have conflicts" and being able to add on any combination of things you want.

Instead, I would recommend the mod author or tool author, take a poll and ask the community what base of mods should be offered to the community, because most likely, it will be a set # of options & combinations.

Well, an a la cart menu that resolves serious conflicts is outside the scope of this project.  It should be sufficient to use a method that handles having the insertion point move unexpectedly (which is what I tried to describe above).  In that example, both mods tried to adjust the elephant's POPULATION_NUMBER and CLUSTER_NUMBER.  The tool would note the multiple changes for the user, with last-in-winning if the merge is allowed to proceed (in this case, Zootastic values would end up in the generated raws).  Note that this is pulling back from my earlier insistence on regular expressions.  We get at least 80% of the usefulness here without turning modders into regex-warriors.

There are a couple other mod manager projects rattling around the forums.  So long as they all share at least one format (which appears to be some kind of diff-from-vanilla), and at least one can grind mod raws into that format, it will all work out in the end.

[thistleknot]

keeping track of insertion points and a cumulative score of line numbers that are removed/added while patching might alleviate the problem with multiple patches.

I was hoping to try to address this very issue with my mod merge tool thread.  Patches by themselves do not report if an object's token's have changed, but merely lines.  I think keeping track of some of the contextual info in the <patch> files would allow one to keep track of object's token's being removed/added for a particular object, maybe even find out if the same area being modified is in conflict due to the same addition/subtraction from another patch having been applied?

So I flattened some of the raws and put them up on github, if anyone wants to work with them.

https://github.com/thistleknot/BasedOnVanillaRaws/blob/40-07-phoebus-flattened/raw/objects/plant_standard.txt

It was recommended I check out octopus merges, and rebasing.

However, this looks promising, "An interdiff is a text file in patch format that describes the changes between two versions of a patch" 

https://www.drupal.org/documentation/git/interdiff

If it does what it sounds like, it would help me see the difference between two diff files that are to be applied?

[King Mir]

The approach I recommend is looking at python's difflib to do a 3 way merge. Then examine each of the merge points for conflicts. Maybe I'll look at this myself sometime.

[PeridexisErrant]

Wow, that's a lot of posts.  Experiments with regexp to flatten files, talk about diff mechanisms... and I haven't even finished a merge-by-overwrite yet!

I'm unlikely to do much for the next few days, busy updating the Starter Pack. 

[hermes]

I thought of something while I didn't have access to my machine.  There are two reasons why N-way merged files break.

Case 1: Mod A adds or deletes lines before mod B tries to change something.  Mod B's changes end up in the wrong place, which may or may not cause issues.

Case 2: Mod A and mod B try to change the same line.

....

This should ensure that changes land where the modder thinks they will based on a vanilla diff.  Won't ensure that the changes are compatible, but at least they won't be off in some other object.

Does that make sense to anyone other than me?

... Which is exactly why I'd go the route of Valdemar's stuff.  First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis.  It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on.  You have two levels of differences - the objects present difference, and the difference between equatable objects themselves.  The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".

In case 1 you'd know that the added lines of B would at least be in the correct object, and if the program was smart it could drop those lines in right after the correct preceding line.

In case 2 you'd know in what object the conflict arose and could prompt the user with relevant information.

This way would make this basic stuff easier, and open the door for more advanced stuff like making connections throughout the raws to entirely remove problem objects.  The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.

The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner.  PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.

[Dirst]

.The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.

I had a feeling that about five minutes thought by a dabbling coder wasn't going to overtake an entire open source community.  Glad that someone has already optimized that idea.

[PeridexisErrant]

I'd go the route of Valdemar's stuff.  First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis.  It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on.  You have two levels of differences - the objects present difference, and the difference between equatable objects themselves.  The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".
<snip>
The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner.  PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.

I should clarify my position a little:

• I am strongly opposed to anything which would require a more complicated input than folders of changed raws, as outlined in OP.  This is because I feel having a simple, usable, standard format is very important for adoption from both users and modders.  If it becomes common to have a readme.txt and configuration.xml - to hold info such as author, name, homepage, base DF version, etc in the latter case - it would be good to use that information but their absence must be handled gracefully.
• I don't want to spend my time on implementing 'smart' merge logic before a basic but working implementation is available (and preferably in the LNP / starter pack, with content created for it).  This is just a matter of priorities for my limited time to work on this, not opposition to the idea as such.
• Merge logic should not present the user with choices beyond the list of mods and load order - if the logic cannot produce a known correct result, the merge should be refused.  This protects users from broken raws; put another way it's the difference between a mod loader which does some merging and a mod merge tool.  Details on what happened could be written to a log file, but probably shouldn't be shown to all users - remember that the target audience is people who are still new to DF and would otherwise avoid mods.
• Other people working on smart merging - so long as it works within the format, as above - is awesome.  I like the discussions about this, think it would be significantly better than a simple diff, and look forward to using it someday.

[hermes]

Merge logic should not present the user with choices beyond the list of mods and load order - if the logic cannot produce a known correct result, the merge should be refused.  This protects users from broken raws; put another way it's the difference between a mod loader which does some merging and a mod merge tool.  Details on what happened could be written to a log file, but probably shouldn't be shown to all users - remember that the target audience is people who are still new to DF and would otherwise avoid mods.

But that's the point!  If any mod removes something from vanilla raws, any other mod, whether applied before or after, could clash with that removal and with a blind diff patch there is no way to guarantee no clash... so you have to refuse the merge.  The diff patch should spot where a mod wanted to edit that removed entry, but it would never detect where that removed entry was referenced somewhere else by another mod.

I went though the exact same thought processes before making my mod loader, and that's why I just went with file overwrite detection.  You either do that, or go all the way and build the data structures.  Half way just lands you in all kinds of problems.

[thistleknot]

could clash with that removal

not if you re-extract base raws with an overwrite, and that's always your input.

Then you work with a base patch system.

I thought about this as well (what if a user modifies his raws?)

One could allow a user to export their current raw set if they so chose, but anytime a mod is applied, it's rewritten over by a base zip

[PeridexisErrant]

No, it could still be a problem.  Imagine vanilla has raws for A, B, and C.  Mod1 modifies A and removes C.  Mod2 doesn't modify A or C - if it did we would refuse the merge - but instead modifies B, which has C as a dependency.  The raws are now broken, unless we have some *really* impressive parsing to catch this kind of thing. 

I don't know how common this might be, but for minor mods it doesn't seem too likely.  I think the best way to deal with this in the short to medium term is just to live with it and reduce the chances by way of major mods, which are more likely to do this, generally being incompatible. 

The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things.  Maybe it just needs to be the full raw folder instead of files changed from vanilla...