Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Dirst]

OK, I stayed up later than I should have and did some more work on the surrounding logic.  I'm happy to call it version 0.2 now, and I've rewritten the readme as well.  Anyone want to test it on some simple or not-so-simple mods?

https://github.com/PeridexisErrant/Py-Mod-Loader

I have an issue with Real Life sucking up a lot of my time lately, but I'll give this a shot when I'm able.  One request though: can we allow the mod to include other folders like Stonesense content?  The only collision should be with the top-level index.txt file.  The logic should allow the eventual inclusion of graphics packs as mods.  I don't use Soundsense (or any sound at all), but if sound packs can be dropped in then those should be allowed as well.

[Button]

OK, I stayed up later than I should have and did some more work on the surrounding logic.  I'm happy to call it version 0.2 now, and I've rewritten the readme as well.  Anyone want to test it on some simple or not-so-simple mods?

https://github.com/PeridexisErrant/Py-Mod-Loader

I don't use Soundsense (or any sound at all), but if sound packs can be dropped in then those should be allowed as well.

Soundsense manages its own sound packs; it doesn't need to be "installed" like graphics packs do.

[IronSI]

PeridexisErrant, I played around a bit with you mod loader.

Notes (Using the following mod packs: All Races Playable, DF Fine Polish, Plantfixes load in that order):

I got a improper indentation error, which I fixed.
I had to figure out whether it was python2 or python3 (Python3). Might want to include a note in the readme to that effect.
I had to figure out where to place the mod files (in LNP/Mods/) Might want to include a note in the readme to that effect.
Upon merging it one set of complete set of raws in temp/raw/ and in temp/raw/objects/, which were different from each other.
Upon merging it initially erased all files in the Plantfixes mod due the folder being named objects in stead of the standard raw.

After deleting the excess raws, I successfully created a pocket embark on a pocket world (With a human civ, no less, thanks to a successfully merged all races playable mod)

I have checked to see if the merges were 'clean' (that is, none of the feature broke anything from the other mods that they shouldn't have) but it appears to works as advertised, but a bit buggy.

If you need any other help PeridexisErrant, let me know and I will be happy to help with my Rusty Competent Python Scripter and my Rusty Competent C Programmer skills.

[PeridexisErrant]

PeridexisErrant, I played around a bit with you mod loader.

Notes (Using the following mod packs: All Races Playable, DF Fine Polish, Plantfixes load in that order):

I got a improper indentation error, which I fixed.
I had to figure out whether it was python2 or python3 (Python3). Might want to include a note in the readme to that effect.
I had to figure out where to place the mod files (in LNP/Mods/) Might want to include a note in the readme to that effect.
Upon merging it one set of complete set of raws in temp/raw/ and in temp/raw/objects/, which were different from each other.
Upon merging it initially erased all files in the Plantfixes mod due the folder being named objects in stead of the standard raw.

After deleting the excess raws, I successfully created a pocket embark on a pocket world (With a human civ, no less, thanks to a successfully merged all races playable mod)

I have checked to see if the merges were 'clean' (that is, none of the feature broke anything from the other mods that they shouldn't have) but it appears to works as advertised, but a bit buggy.

If you need any other help PeridexisErrant, let me know and I will be happy to help with my Rusty Competent Python Scripter and my Rusty Competent C Programmer skills.

Awesome!  Thanks for the feedback.  I'll clarify the usage and version information in the readme. 

I think your problems with duplicate raws and the plantfixes mod have the same cause:  a mod used as input that wasn't in the expected format.  It sounds like the plantfixes mod is distributed as an 'objects' folder, which you renamed 'raw' - this would cause the duplication, as the objects files don't exist in the top-level folder in vanilla (and deleting them would also effectively remove that mod).  Try putting them in "LNP/Mods/Plantfixes/raw" (/objects/files) instead of renaming.  Remember that the input must be structured so you could drop the contents of the mods folder over the DF folder to install it.  This does make me think that checking the validity of the inputs should take a higher priority though - ideally it could detect and fix this kind of issue, and warn you in more complicated cases. 

Without any information, I'll assume that the improper indentation error was caused by indentation in one of the mods you merged.  If the mod changed indentation, it could be valid on it's own but not once merged. 

[PeridexisErrant]

Hi all - I've posted a full folder structure with nine mods included and formatted as an example. 

Here it is:  http://dffd.wimbli.com/file.php?id=9428

[IronSI]

The indentation error was in your python code, one of the triple quoted blocks was indented too much (Lines 212 and 213).

Upon review of the raw folders (putting everything in into raw/objects folders), it doesn't appear to be cause by the Plantfixes mod, but the Df-Fine-Polish has subfolders within the objects folder, would that affect anything?

[PeridexisErrant]

Oops, I fixed that in the download above. 

Yes, any variation from the vanilla file structure will mean files have to be copied instead of merged - which massively increases the chance of conflicts.  You can't really avoid new files with some mods, but trying to 'fix' the folder structure should make them easier to spot. 

[Dirst]

Yes, any variation from the vanilla file structure will mean files have to be copied instead of merged - which massively increases the chance of conflicts.  You can't really avoid new files with some mods, but trying to 'fix' the folder structure should make them easier to spot.

This looks like a very good start, though I'm not sure why it couldn't handle stuff outside the raw folder (like Stonesense XML or a new embark profile) so long as it doesn't attempt any flattening there.

I tried to look through Python's documentation to see if I could be of any more help on the project, but it looks like it will be a while before I do anything other than tweak other people's code.  I did, however, see that Python has a nice ElementTree structure that can be used to hold (among other things) file directory information or XML data.  Not sure if an ElementTree would be easier than shutil, but the XML parser massively simplifies the problem of using a manifest file.

Code: [Select]

<?xml version="1.0" ?>

<!-- This XML contains enough information to make an educated guess about how two mods would merge *without processing any files.*
     If it sees multiple mods affecting the same file, it notifies the user that it has to try a merge to figure out compatibility.
     This means that the tool can download a bunch of these manifests in bulk to preview mods.  -->

<modpack token="Example"> <!-- The token is the name of the folder holding the mod, and how mods refer to one another. -->
<title>The Example Mod</title>
<version>0</version><subversion>90</subversion> <!-- Would display as "v0.90".  Can also add a <revision> tag for "v0.90r2" -->
<author>Dirst</author>
<thread>http://www.bay12forums.com/smf/index.php?topic=000000</thread>
<download>http://dffd.wimbli.com/file.php?id=0000</download>
<description>This mod is an example used to illustrate the optional manifest file.</description>
<dependencies>
<dfhack/>
<mod token="Basic" min_version="0" min_subversion="01" order="before"/>
</dependencies>
<compatibility>
<core max_version="40" max_subversion="07">  <!-- The lack of min_ will match anything v40.07 or earlier. -->
<color>red</color>
<message level="3">The Example Mod is not compatible with Dwarf Fortress before v40.08.</message>
</core>
<core min_version="40" min_subversion="08" max_version="40" max_subversion="11">
<color>green</color>
</core>
<core min_version"40" min_subversion="12">
<color>orange</color>
<message level="3">The Example Mod has not been tested with this version of Dwarf Fortress.</message>
</core>
<dfhack min_version="40" min_subversion="08" min_revision="2" max_version="40" max_subversion="10" max_revision="1">
<color>green</color>
<message level="1">"The Example Mod" employs DFHack's interactionTrigger and lua scripts.</message>
</dfhack>
<dfhack min_version="40" min_subversion="10" min_revision="2">
<color>orange</color>
<message level="3">The Example Mod has not been tested with this version of DFHack.</message>
<message level="1">"The Example Mod" employs DFHack's interactionTrigger and lua scripts.</message>
</dfhack>
<mod token="CLA" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="Ironhand" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="MasterworkDF"> <!-- Lack of min_ and max_ will match any version -->
<color>red</color>
<message level="3">The Example Mod is known to be incompatible with Masterwork DF.</message>
</mod>
<mod token="Mayday" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="OldGenesis">
<color>red</color>
<message level="3">The Example Mod is known to be incompatible with Old Genesis.</message>
</mod>
<mod token="Phoebus" min_version="40" min_subversion="07" max_version="40">
<color>green</color>
</mod>
<mod token="Spacefox" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="Other" min_version="1" min_subversion="0" max_version="1" max_subversion="35">
<color>yellow</color>
<message level="2">There have been no reports of conflicts with this version of That Other Mod.</message>
<message level="1">"The Example Mod" & "That Other Mod" both add to entity_default.txt, but do not appear to affect one another's edits.</message>
</mod>
<mod token="Basic" min_version="0" min_subversion="01" max_version="0" max_subversion="20">
<color>green</color>
<message level="1">"The Example Mod" is an extension of "The Basic Mod" and will not function without it.</message>
</mod>
<mod token="Basic" min_version="0" min_subversion="21">
<color>orange</color>
<message level="3">"The Example Mod" has not been tested with this version of "The Basic Mod".</message>
<message level="1">"The Example Mod" is an extension of "The Basic Mod" and will not function without it.</message>
</mod>
</compatibility>
<manifest>
<!-- Upon merge, any files with no path info are copied into an LNP documentation subfolder named after the token. -->
<!-- The parse attribute defaults to "flatten" which also strips out anything outside of []s. -->
<file parse="text">manifest.xml</file> <!-- Encode linefeeds if needed, but do not flatten this file. -->
<file parse="text">readme.txt</file>
<file parse="none">The Example Mod v0.90.pdf</file> <!-- Pass this file exactly as-is. -->
<file parse="none">data/art/font.TTF</file>
<file>data/init/embark_profiles.txt</file> <!-- Should be able to add a profile to the end of the user's set. -->
<file parse="text">data/init/overrides.txt</file> <!-- TWBT uses # to comment out inactive lines, so can't trust flattening. -->
<file parse="text">raw/onLoad.init</file> <!-- Mods that use DFHack will probably want to add lines to this file. -->
<file>raw/graphics/graphics_example.txt</file>
<file parse="none">raw/graphics/example/example.png</file>
<file>raw/objects/creature_example.txt</file>
<file>raw/objects/entity_default.txt</file> <!-- Many mods will need to add lines into this file. -->
<file>raw/objects/interaction_example.txt</file>
<file>raw/objects/reaction_example.txt</file>
<file parse="text">raw/objects/text/secret_wisdom.txt</file>
<file parse="text">raw/scripts/example-script.lua</file>
<file parse="text">stonesense/index.txt</file> <!-- Any Stonesense content will need to add a line to this file. -->
<file parse="text">stonesense/example/index.txt</file>
<file parse="none">stonesense/example/example.png</file>
<file parse="text">stonesense/example/example.xml</file>
</manifest>
</modpack>

The first part contains some identifying information about the mod as it might appear in a GUI loader.  A set of <dependencies> lets the tool know if DFHack or any other mod is required for this mod to function.  A structure of <compatibility> data allows for hand-crafted overrides for specific combinations of mods.  This can be displayed to the user even before a merge is attempted.  The actual <manifest> at the end identifies all of the files.  We could hardcode how certain folders and certain extensions are handled, but this seems more future-proof.

One issue with <compatibility> is that there could be zero, one or two bits of XML that are relevant:
0: No pre-merge feedback to the user because these two mods have never heard of each other before.
1: Trust the override that is present and display the message in the GUI before the merge is attempted.  The minimum message level is user configurable.  All messages might get sent to a log.
2: Color the mods according to the "redder" data, and present both sets messages to the user for him/her to decide.  Maybe cull duplicate identical messages.

The manifest is the only thing that a modder would be asked to do specifically for the loader, and even that is optional.

Edit: Fixed missing word and added link to Python docs on ElementTree.

[Button]

Hrm. I was sort of hoping for a way to put more... granular merging instructions into a manifest file. For example, take my vampire fodder minor mod (one of the modular modpack I uploaded for testing), which replaces a number of creatures' BLOOD or ICHOR with a custom BLOOD2 or ICHOR2. It would be super sweet if I could tell the merger that, if another mod messes with BLOOD for [these creature objects], please change it so it messes with BLOOD2 instead.

Now that I write that out, though, it sounds unrealistically ambitious, so never mind.

[Dirst]

Hrm. I was sort of hoping for a way to put more... granular merging instructions into a manifest file. For example, take my vampire fodder minor mod (one of the modular modpack I uploaded for testing), which replaces a number of creatures' BLOOD or ICHOR with a custom BLOOD2 or ICHOR2. It would be super sweet if I could tell the merger that, if another mod messes with BLOOD for [these creature objects], please change it so it messes with BLOOD2 instead.

Now that I write that out, though, it sounds unrealistically ambitious, so never mind.

The goal is to parse things that could individually function if just dropped onto the DF folder.  Sometime after v1.0, it might make sense to fold in something more elaborate such as scripting.  But I think it'd be easier to user Rubble as a back end for that rather than re-inventing the wheel.

[Button]

Skimmed over your script and the logic looks OK, but it's not really runnable right now >>. Herp derp it's supposed to be Python 3.


I think your observation about mods which put new entities in new files being more error-prone than those which put new entities in existing files is flawed. AFAICT the reason you think this is that you don't have any dupe-checking logic between raw files - but dupes could just as easily be placed in separate vanilla raw files as they could in separate modded raw files.

[PeridexisErrant]

Skimmed over your script and the logic looks OK, but it's not really runnable right now >>. Herp derp it's supposed to be Python 3.


I think your observation about mods which put new entities in new files being more error-prone than those which put new entities in existing files is flawed. AFAICT the reason you think this is that you don't have any dupe-checking logic between raw files - but dupes could just as easily be placed in separate vanilla raw files as they could in separate modded raw files.

I do intend to make it usable in 2.7, but haven't gotten around to it yet. 

Duplication checking would certainly help, but it's not the main issue.  Basically it treats adding files as a proxy for "this is a major mod, and therefore more likely to cause problems".  There's also a few edge cases like "Mod A adds creatures (in a new file), Mod B removes all creatures (but silently misses those in the new file)" - adding creatures in a block removed by mod B would be caught. 

[PeridexisErrant]

Concept image: 


Click mod in bottom list to append to top list, click mod in top list to remove from top list.  Once you're happy with some combination of green and yellow, click install! 

The launcher rebuilds the merge every time the top list changes and provides live feedback via the colours (yes, it's that fast).  The logic is described higher in the thread, basically line-by-line changes where a second change from the common ancestor (vanilla) is rejected. 

Until some more of the kinks get worked out, this will be hidden unless you enable it in the LNP options file (hopefully). 


Next steps, in no particular order: 

• Implement the GUI
• Define a manifest file ("manifest.json")
• What goes in it?  I'm thinking at least mod version string, for DF version ___, author, homepage, source (was the manifest generated or written?), short description (140 chars, so it's tool-tippable)
• Start using the manifest, or at least generate a best-guess if it's not present
• Better handling of graphics, save conflicts, etc.
• Write a file in the generated raw folder listing the mods merged, so people can check later

[PeridexisErrant]

Exciting news all - with help from Pidgeot, I got the mod merger working as part of a PyLNP fork!
With a working GUI, functional merge logic, and the input format all sorted out I think it's time to close this thread. 

Please join me in the new thread! 

It's going to be a party over there, with refinements to the logic and interface - and a call going out for suitable mods to be included 
Time to strike the mods!

Spoiler: Pics of it working! (click to show/hide)

Not seen:  green, for mods which are entirely within vanilla files and merged OK. 
Yellow mods add files, and were merged successfully.  All these would be included if you hit 'install'. 
Red is a rejected merge due to overlapping changes. 
Orange was not attempted due to a previous conflict.