Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[thistleknot]

I have an idea to start on this project.  So far I only have the ability to load two text files and derive a google-diff-match-patch here...

• https://github.com/thistleknot/DiffMatchPatch

Spoiler: to learn how to use QT creator (click to show/hide)

I highly recommend these two tutorials back to back, took me maybe 2 days to get up and running with layouts and how to get menu items to operate and load text files and output to text.

• http://qt-project.org/doc/qt-5/gettingstartedqt.html
• http://qt-project.org/doc/qt-5/designer-layouts.html

invoking new forms [after creating them with qt's form editor]:

• http://stackoverflow.com/questions/1518317/how-to-show-another-window-from-mainwindow-in-qt

5.3.1 Offline Installer:

http://download.qt-project.org/official_releases/qt/5.3/5.3.1/qt-opensource-windows-x86-mingw482_opengl-5.3.1.exe

but...

I now have an idea of what I actually want to do versus just make another winmerge, tortoisegitmerge, or basically just another patch program.

• I want to keep track of tokens in an object, and merge based on token additions or subtractions.

I have no idea how long it would take me to do something like that, but I started making some skeletal ideas on what I would need to do to accomplish that.  So far the idea is to parse a bunch of tokens (maybe find missing closing brackets).

The idea would be to mimic rawexplorer object loading and mod manager's merge capabilities.  Mod Manager did an ok job of merging using patch logic, but I want to do it based on a: by object by token level, and insert/delete tokens based on what the mod is bringing.

This tool hopefully would allow people to merely derive patches between two versions of the game [raw's specifically], and then apply that patch to another mod, no matter what version of that mod is.  Lofty goal I know.

The mod's in question will be based on pre-derived patch files.  I could use patch files as is, but it would be far better to also log the object that is being modded ahead of time and put that in the patch file for easier patch application.

I hope to use this thread as a starting point for further discussion on such a project, basically a systems requirements document if you will, maybe get a collective project going.

[thistleknot]

**reserved**

[YAHG]

This makes me think of the modular modding that CIV5 defaulted to. In my head we are using mod patch files so something like:

[FIND:GOBLIN]
[ADDTAG:NATURALSKILL:DISCIPLINE:1]
[END:GOBLIN]
[FIND:TROLL:MALE]
[ADDTAG:NOFEAR]
[GOTONEXT:SELECT*]
[REMOVE:GAITCLIMBER*] to the next "]" with some unknown stuff in the middle and then remove it all is the idea I'm imagining.


Is that the sorta thing you are thinking? Or like something smarter or whatever ?

[thistleknot]

yes.

I don't know if the position of a token within the object matters (I kind of think it does especially with things like castes), but I have to figure out how to implement the token at the right spot based on prior token positioning.  And hopefully not mess up the original file too much so it doesn't break regular patching behavior.

So here's a patch file:

Code: [Select]

@@ -27,6 +27,12 @@ b_detail_plan_default
[ADD_MATERIAL:LEATHER:CREATURE_MAT:ANIMAL_LEATHER:LEATHER]
[ADD_MATERIAL:TALLOW:TALLOW_TEMPLATE]
[ADD_MATERIAL:SOAP:SOAP_TEMPLATE]
+? ###Tiered Leather
+ [ADD_MATERIAL:LEATHER_2:PADDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_3:STUDDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_4:PLATED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_5:RUNIC_LEATHER_TEMPLATE]
+ ###End Teared Leather
[BODY_DETAIL_PLAN:CHITIN_MATERIALS]
[ADD_MATERIAL:CHITIN:CHITIN_TEMPLATE]

What I would do is keep track of the object that was being modified as well, so the patch file would have an object specifier.

in this case, the patch file is specifically modifiying this object:

[OBJECT:BODY_DETAIL_PLAN]

[BODY_DETAIL_PLAN:STANDARD_MATERIALS]

so those would be pre-pended before the actual patch information.

Then we'd jump to the object in question in the file to be patched, read the tokens in the object in it's entirety [preferably in it's own QPlainTextEdit box], and try to find the specific area of the object to "patch" by finding the before and after tokens
before tokens

Before Tokens:

Code: [Select]

[ADD_MATERIAL:LEATHER:CREATURE_MAT:ANIMAL_LEATHER:LEATHER]
[ADD_MATERIAL:TALLOW:TALLOW_TEMPLATE]
[ADD_MATERIAL:SOAP:SOAP_TEMPLATE]

After tokens

Code: [Select]

[BODY_DETAIL_PLAN:CHITIN_MATERIALS]
[ADD_MATERIAL:CHITIN:CHITIN_TEMPLATE]

and insert these tokens inbetween

Code: [Select]

+ [ADD_MATERIAL:LEATHER_2:PADDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_3:STUDDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_4:PLATED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_5:RUNIC_LEATHER_TEMPLATE]

If the before and after don't match in exact order, we double check the rest of the object's tokens for pattern recognition of these matching [before and after] tokens, maybe using google's diff patch match to do so, or simply a simple count of the # of matching tokens to identify the proper before/after position to inject the changed tokens into.

Note:
We also have to check if any of these before/after tokens are tokens that we are adding.  If they are, then some sort of decision needs to be made to cancel a token out.

Obviously, we'd have to be sure we're removing the right tokens to.  That could require it's counting algorithm.

Some obvious issues is how to deal with comments.  I think mod manager couldn't handle them, but things like comments can be treated as behavior tied to the token that they are attached to when adding/removing tokens.

[Dirst]

Before I get started, is this the same project that the Starter Pack folks mentioned about using diffs for mods?  Either way, my thoughts on what's here so far:

I don't know if the position of a token within the object matters (I kind of think it does especially with things like castes), but I have to figure out how to implement the token at the right spot based on prior token positioning.  And hopefully not mess up the original file too much so it doesn't break regular patching behavior.

It can be critical for creature-related stuff.  I'd just assume position is important unless the modder indicates otherwise.

So here's a patch file:

Code: [Select]

@@ -27,6 +27,12 @@ b_detail_plan_default
[ADD_MATERIAL:LEATHER:CREATURE_MAT:ANIMAL_LEATHER:LEATHER]
[ADD_MATERIAL:TALLOW:TALLOW_TEMPLATE]
[ADD_MATERIAL:SOAP:SOAP_TEMPLATE]
+? ###Tiered Leather
+ [ADD_MATERIAL:LEATHER_2:PADDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_3:STUDDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_4:PLATED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_5:RUNIC_LEATHER_TEMPLATE]
+ ###End Teared Leather
[BODY_DETAIL_PLAN:CHITIN_MATERIALS]
[ADD_MATERIAL:CHITIN:CHITIN_TEMPLATE]

What I would do is keep track of the object that was being modified as well, so the patch file would have an object specifier.

in this case, the patch file is specifically modifiying this object:

[OBJECT:BODY_DETAIL_PLAN]

[BODY_DETAIL_PLAN:STANDARD_MATERIALS]

so those would be pre-pended before the actual patch information.

You lost me there.  Is that object info encoded in the @@ line at the top?  I'd much rather specify the object by type and token, just in case some other mod has left things untidy.  [BODY_DETAIL_PLAN:STANDARD_MATERIALS] is enough to tell the program to look in body detail plan files.  Just need some pseudo-objects to handle init files and such.  Ideally, an entire mod could be contained in a single file.

Then we'd jump to the object in question in the file to be patched, read the tokens in the object in it's entirety [preferably in it's own QPlainTextEdit box], and try to find the specific area of the object to "patch" by finding the before and after tokens
before tokens

Before Tokens:

Code: [Select]

[ADD_MATERIAL:LEATHER:CREATURE_MAT:ANIMAL_LEATHER:LEATHER]
[ADD_MATERIAL:TALLOW:TALLOW_TEMPLATE]
[ADD_MATERIAL:SOAP:SOAP_TEMPLATE]

After tokens

Code: [Select]

[BODY_DETAIL_PLAN:CHITIN_MATERIALS]
[ADD_MATERIAL:CHITIN:CHITIN_TEMPLATE]

and insert these tokens inbetween

Code: [Select]

+ [ADD_MATERIAL:LEATHER_2:PADDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_3:STUDDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_4:PLATED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_5:RUNIC_LEATHER_TEMPLATE]

If the before and after don't match in exact order, we double check the rest of the object's tokens for pattern recognition of these matching [before and after] tokens, maybe using google's diff patch match to do so, or simply a simple count of the # of matching tokens to identify the proper before/after position to inject the changed tokens into.

Again, it'd be best not to assume you're starting from vanilla, especially since I've seen graphics packs that re-arrange the raws for internal consistency.  Some kind of pattern matching (wildcards or even regular expressions) would help, and the parser should pre-process both files to virtually strip out comments and put each token on its own line.  The order of the tokens shouldn't matter either, the parser just looks until it has matched each "before" tag, and places the "cursor" just before the "[" after the last match.

Similar logic applies to the "after" tags (with a hardcoded condition that all matches need to occur before the next object is defined).  These "after" tags ought to be optional.  I'm thinking that in the case of overlap, it's more important to be after the "before" than before the "after".

Note:
We also have to check if any of these before/after tokens are tokens that we are adding.  If they are, then some sort of decision needs to be made to cancel a token out.

The precedent with CREATURE_VARIATION raws is to make it the modder's responsibility to delete a tag before adding it back.  There is, however, a syntax for fiddling with the parameters within a tag.  Might be worth looking into replicating that somehow.

If regular expressions are used to perform the positioning, it might be useful to someone to re-use the wildcard matches within their injected tokens.  Can't think of a use case off the top of my head, but it might come along for free with a canned regular expression parser.

Obviously, we'd have to be sure we're removing the right tokens to.  That could require it's counting algorithm.

Some obvious issues is how to deal with comments.  I think mod manager couldn't handle them, but things like comments can be treated as behavior tied to the token that they are attached to when adding/removing tokens.

The pattern in vanilla seems to be comments on the same line as tokens apply to the last token, whereas comments on their own line(s) apply to the next token.  That should handle the vast majority of comment cases.  The two exceptions I remember are (1) giving beards to female Dwarves has comments before a line that you could choose to make into a token by adding brackets, and (2) the Starter Pack will switch some brackets into exclamation points to disable certain tags.

That brings up one other issue: possibly-existing tags.  Maybe just start a before or after token with a question-mark (or other special character) to indicate that it's used for positioning if it exists, but ignored otherwise.

[thistleknot]

Before I get started, is this the same project that the Starter Pack folks mentioned about using diffs for mods?  Either way, my thoughts on what's here so far:

You lost me there.  Is that object info encoded in the @@ line at the top?  I'd much rather specify the object by type and token, just in case some other mod has left things untidy.  [BODY_DETAIL_PLAN:STANDARD_MATERIALS] is enough to tell the program to look in body detail plan files.  Just need some pseudo-objects to handle init files and such.  Ideally, an entire mod could be contained in a single file.

...


Again, it'd be best not to assume you're starting from vanilla, especially since I've seen graphics packs that re-arrange the raws for internal consistency. 

I got this idea from github, I don't know what other projects people are working on, Fricy mentioned something in a MW poll about the direction new modding should head in, I had this idea from back when I used to fail using mod manager, but recently learned the concept of cherry picking patch methods.

I don't know why your confused.  The patch file was a LITERAL unified patch file, I was just supplying it as an example, WHAT IT IS MISSING is the object identifiers such as


[OBJECT:BODY_DETAIL_PLAN]

[BODY_DETAIL_PLAN:STANDARD_MATERIALS]

that I would PREPEND to the front of the patch file.

I never said I'd start with vanilla [okay, maybe I said I'd use vanilla as a base].  What the goal is... is to identify differences FROM vanilla.  That way a patch can be applied to "vanilla" dwarf fortress and turn it into a mod.

However, it doesn't have to be based on vanilla, but if one wants the exact patch differences between say a mod and vanilla. that would be what needs to be extracted to be applied to another mod.

Say I compare civforge to vanilla.

then I want to extract just what civforge changes from vanilla

and then import it over another mod

but... I think the same concept could be applied from the differences between civforge and say genesis... it's just that's what the patch is going to be based on... so if one applies it to vanilla, it's going to get you weird results I think.

The way I have github setup, is I handle all mod changes from a base, and I set the base as vanilla so I can see what that mod did differently.  However, I'm not sure what I would base it on otherwise.

I had a different idea with comments, I don't want to assume they need to be on their own line.

I want to just position them where they were before in relation to tags/tokens.

If it was inbetween two tags, then put it back inbetween two tags.

If it was on it's own line, put it back on it's own line.

TBH, i'm such a newb with this, I'm still trying to get the hang of outputting to a new window before I start talking too much indepth of parsing methods.

[Meph]

Yes, best you speak with Putnam, before both of you start something similar. He sounded very enthusiastic about the idea of a Mod Start Pack, which does essentially what you describe here. Vanilla + optional mods.

[Roses]

Now that RawExplorer is moving to an Open Source system, I wonder if it wouldn't be easier to start from there and add the merging functionality into it as a plugin, since the reading and parsing and displaying of the raws is already handled so nicely.

[Dirst]

Before I get started, is this the same project that the Starter Pack folks mentioned about using diffs for mods?  Either way, my thoughts on what's here so far:

You lost me there.  Is that object info encoded in the @@ line at the top?  I'd much rather specify the object by type and token, just in case some other mod has left things untidy.  [BODY_DETAIL_PLAN:STANDARD_MATERIALS] is enough to tell the program to look in body detail plan files.  Just need some pseudo-objects to handle init files and such.  Ideally, an entire mod could be contained in a single file.

I got this idea from github, I don't know what other projects people are working on.

I don't know why your confused.  The patch file was a LITERAL unified patch file, I was just supplying it as an example, WHAT IT IS MISSING is the object identifiers such as


[OBJECT:BODY_DETAIL_PLAN]

[BODY_DETAIL_PLAN:STANDARD_MATERIALS]

that I would PREPEND to the front of the patch file.

I'm confused because I've done raw modding, but haven't done any kind of heavy-duty coding that actually uses diff files before.  So consider this an outsider's view, and feel free to ignore if this doesn't make sense

I was picturing something like

Code: [Select]

= [BODY_DETAIL_PLAN:STANDARD_MATERIALS] (indicates the object to work on)
[ADD_MATERIAL:LEATHER:CREATURE_MAT:ANIMAL_LEATHER:LEATHER]
[ADD_MATERIAL:TALLOW:TALLOW_TEMPLATE]
[ADD_MATERIAL:SOAP:SOAP_TEMPLATE]
+? ###Tiered Leather
+ [ADD_MATERIAL:LEATHER_2:PADDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_3:STUDDED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_4:PLATED_LEATHER_TEMPLATE]
+ [ADD_MATERIAL:LEATHER_5:RUNIC_LEATHER_TEMPLATE]
+ ###End Tiered Leather
[BODY_DETAIL_PLAN:CHITIN_MATERIALS]
[ADD_MATERIAL:CHITIN:CHITIN_TEMPLATE]

= [ENTITY:MOUNTAIN]
[PERMITTED_BUILDING:SOAP_MAKER]
+ [PERMITTED_BUILDING:BATHHOUSE]

= [CREATURE:TIGER]
* \[GRASSTRAMPLE:[0-9]*?\] (indicates the line is to be considered a regular expression, note that brackets need to be backslashed)
? [PREFSTRING:stripes, of course] (indicates that the line should be considered for position, but only if it exists)
+ [PERFSTRING:hunting prowess]

= [CREATURE:ELEPHANT]
[NATURAL]
?* \[PREFSTRING:.*?\] (this line is a pattern match AND optional)
+ [PREFSTRING:trunks]

Am I making any sense?  Of course, all of this could be moot from what Meph just said.

[thistleknot]

yeah, well. if someone else beats me to the punch (and is a better programmer like Putnam), then so be it.

I'm just trying to practice with QT and I figured that this would be a good start.  Originally I just wanted to create a front-end tool for google-diff-match-patch, but then I started thinking about what I could in the long term.

Rawexplorer would be a good base... but I think it's built in python, same with mod manager.  Either way, it's not in QT.

If anything, I might just port some functionality, but I have a desire to work with QT--because it's QT--that I want to add to my resume [if I can].

I did get the idea of the patches from github, now that I have this token concept vs using patch files as is, I think it can be a good step forward (for anyone who wants to take the helm up).

The patch file you drew up is basically the idea I had; yes, Dirst.

Tbh, I'd MUCH RATHER modify the source of mod manager vs RawExplorer, since Mod Manager already implements a type of patch logic, if I wanted to do this quickly and outside of QT, i'd use mod manager as a base, then maybe borrow some code from rawexplorer.  But the two do pretty much the exact same thing, but I think rawexplorer allows one to alphabetize (which is a function that I could see needing to be used to determine if an object already exists in a mod).

Update:
I notified Putnam that I don't intend on redoing any work he is working on.

[PeridexisErrant]

PTW, this looks nice. 

Now all the community needs is a standard format for patch-based mods, and they'll take off!

[Putnam]

PTW, this looks nice. 

Now all the community needs is a standard format for patch-based mods, and they'll take off!

A "standard patch format" could easily be made to simply be... the raws.

Also, the thought of me being a better programmer is laughable, heh. I haven't done any work on an actual merging program, just some work on prettifying Warmist's mod-manager, but a text-based UI will never be pretty enough for some people.

[thistleknot]

one potential problem I see with a token/tag based mod merger.

Is take the tigerman.

v34.11 vs v40.00+

So many tokens were changed inbetween the two versions, it might be hard to identify the right object...

OR WORSE.

Both mods had a completely different object by the same name.

Yeah, I don't know how to handle a situation like that other than trying to merge the two and/or notifying a player in advance before doing so.

[Putnam]

Check if there's an existing one and rename the new one to (x)a or something.

[Dirst]

Check if there's an existing one and rename the new one to (x)a or something.

I think it's reasonable to assign some responsibility to the person using the tool.  Maybe two mods were trying to adjust the same object (two attempts at turning fanciful CENTAURs into actual creatures), maybe one is trying to tweak the other's changes (fiddling with the changes made by a graphics pack), or two different people happen to have come up with the same idea for a name (a HEARTH building, a CAVEMAN entity, etc.).  It would be nigh impossible for the tool to figure out which case is which.  Bonus points if the user can assign prefixes in that third case without re-writing the source files.