Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[PeridexisErrant]

I hope I haven't killed this thread/discouraged anybody from working on it. As you may have noticed I don't have a ton of time to work on this myself so please don't think that I've claimed the project or anything.

"I don't have a ton of time to work on this" basically sums up my position, except I'm also bumping up against the limits of my programming ability.  My sneaky plan is to wait until my semester course covers something relevant, and then retry - but if someone else finishes first I'm not going to be sad!

I know everyone is talking about automation and the best script methods to merge mods together, but why not have one person do it by hand? I have done it dozens of times, and its quite easy if you take minor mods and vanilla DF as base. Since you say that you don't even want to try MDF + Genesis + LfR or something similar, it might be that the entire process of writing the code to merge mods is much more work than merging them by hand.

I mean... this thread is over 2 weeks old. In that time I could have merged 100 minor mods into a modpack, using simple on/off toggles on the [OBJECT:X] line at the top, and using specific filenames, with only the current visual-basics based LNP/Starter Pack GUI as base. And I dont have coding knowledge, I would copy+paste the AQUIFER button you have for this.

I dont mean to intrude, but I have the feeling that you are overcomplicating stuff. Although I have to add that new additions would require recompiling the launcher. Splinterz would probably find a way around that, even using the system I described, by using a dropdown menu and an external txt that modders could add their mods to, he did the same for MDF.

Suggestion: Add a new tab to your normal DF Starter Pack, with 10 very popular minor mods, and wait for peoples feedback? That's what I would do.

The medium-term goal is to add the mod loader as a new tab in the PyLNP, which I'm switching over to soon.  That will mean that the format is a standard available to new players on whatever OS, which should help adoption from users and modders.  Recompiling isn't much of an issue once the source code has been written.  For a while it might be a binary choice between mods and the ability to change graphics, but I'm hoping we can solve that eventually. 

You're right that a simple mod pack could have been assembled in less time, but the goal of this isn't just to make a mod pack available - we already have those.  I want to help new players get into mods, and that means they need to be able to add new mods to the pack as easily as anyone can add utilities or graphics to the LNP.  It's likely that we'll compile a basic content set for this - making some popular small mods as minimal as possible to reduce conflicts and so on - but that's not meant to be a core part of the program, which should be flexible enough to use whatever content is fed to it without having to be rebuilt each time. 

We may well be overcomplicating things, but I'm feeling ambitious about the long term. It's important to note that I've designed something that works at a basic level for the merging, which we can improve the logic of later.  Unfortunately I can't work out how to apply a unified diff with Python, or I'd have had a working implementation a week ago...

[Meph]

Ah, so you want a random person who has this Mod Pack to download any mod he likes from DFFD, drag+drop it into some folder, and have the Mod Pack automatically add it to its list of available modules that it installs/deinstalls. Thats a pretty neat idea.

Only downside I see is that it requires more initiative than I attribute to the average currently-non-mod-using player. It would still require an active mindset of looking through the forum, seeing a mod they like by finding it in the modding board and reading the description, going to DFFD, download it, unpack it, drag+drop it into the Mod Loader. Reminds me of Nexus Mods, Oblivion or Skyrim. ^^

And cross compability for linux/mac is a big plus of course.

[thistleknot]

Yes its ugly. And primitive. But you know that only because you have modding/programming background knowledge. Users dont care.

I think Meph's way is kind of an intelligent post processing of raws.

I recommended a patch file that injected a post processor ###Flag### [based on how Meph has his various flagged raws load] w while back in this thread.

I also do think we are over-complicating what could be accomplished with just patching and scripting.  However, each mod would be static in such a scenario.

Trying to build a "merge" of mods... I'm not sure how Meph would go about resolving that with manual edits.  It would involve a method to track the merging of various mods and there affect on multiple tokens...  unless somehow that system already incorporated some sort of "classification" system for the ###Flags.

However, the only way to do it without "coding" is to do it by hand; which takes away have the fun imo.

Whatever you derive by hand, at least the batch file I derived for processing raw's, would keep your ###Token's intact.

On further thought.

Your mod loader has code that parses tokens based on these "###Token's###"... what language is it in?

[Meph]

Its Visual Basics. It literally just goes through text files, looks for YES_SOMETHING[ and replaces it with NO_SOMETHING!, removing the [ , which makes DF not accept the tag as correct. It requires .NET-framework and only works on Windows, so its not useable for what Peridexis and co. are talking about here.

[PeridexisErrant]

Ah, so you want a random person who has this Mod Pack to download any mod he likes from DFFD, drag+drop it into some folder, and have the Mod Pack automatically add it to its list of available modules that it installs/deinstalls. Thats a pretty neat idea.

Only downside I see is that it requires more initiative than I attribute to the average currently-non-mod-using player. It would still require an active mindset of looking through the forum, seeing a mod they like by finding it in the modding board and reading the description, going to DFFD, download it, unpack it, drag+drop it into the Mod Loader. Reminds me of Nexus Mods, Oblivion or Skyrim. ^^

And cross compability for linux/mac is a big plus of course.

Yep, that's it exactly   

We can't make it entirely painless, but any improvement is good and this could be pretty big.  Many players will never go past the defaults, but if ten percent decide to add something that's still a serious influx of mod-users for the currently-neglected small mods.  The easy-add feature also makes it easier for whoever maintains and distributes the pack. 

My concept for the way load order works is uh, very similar, to the elder scrolls model - user chooses order, gets feedback on compatibility, retry until it works or you realise it's not going to work at all.  Early implementations should err on the side of rejecting merges that might be OK, but it should get smarter over time.  One important longer-term goal - which should be useful more generally - is to eventually parse and assess the internal state of the raws; and notify the user of problems with broken tags, references to nothing, missing dependencies, broken templates, and so on.  Like the rest, this will probably start very primitive and get upgraded over time. 

The search-and-replace would be easy to implement on a pre-compiled set of raws for however many mods (though tedious to do very many), but I can't see a robust way to add new content in the drag-and-drop way we're targeting. 

[thistleknot]

the 2 weeks of conversation is due to

My concept for the way load order works is uh, very similar, to the elder scrolls model - user chooses order, gets feedback on compatibility, retry until it works or you realise it's not going to work at all

you're talking about a mod tool for another game in who knows what language that can't just be "ported" over to DF.

So... I would propose something simple first.

v1.0
Static Mods (scripts, parsed raws [either python parsed or batch/shell parsed], and patch files)

Then...

v2.0
Do a mod merger (maybe using a system similar to Meph's search/replace concept; mods that don't merge though, would have to be figured out somehow so the proper flags wouldn't conflict)

Then maybe an

v3.0
advanced object merger.

We're waiting for a mansion when all we need is a shack atm.

We literally have ALL the tools to create a batch based mod system.  User clicks on a batch file or linux shell script, then gets a bunch of text prompts asking what mod he wants to load.  Then the mod is applied, just before the game is started from the gui or batch file.

We could EVEN HOST A GITHUB repository of all the mods so a user can go and download them

Hell, we could use github to create our PATCH files for us.

BTW, this is just a proof of concept, but it could easily be done rather than having a user have to hunt mods down and parse and merge.

However, in the meantime, I'd propose just including a few mods, say 5 to 10 in an initial release, and work on a repo where users can upload their mods.  Hell, the theory is easy enough for anyone to implements.  The batch file could simply list a subfolder's dir contents where the mods are stored.... and parse from their.  The batch file would automagically parse and patch the mod compared to a base vanilla version, which is entirely possible using diff3.

[Dirst]

the 2 weeks of conversation is due to

My concept for the way load order works is uh, very similar, to the elder scrolls model - user chooses order, gets feedback on compatibility, retry until it works or you realise it's not going to work at all

you're talking about a mod tool for another game in who knows what language that can't just be "ported" over to DF.

So... I would propose something simple first.

v1.0
Static Mods (scripts, parsed raws [either python parsed or batch/shell parsed], and patch files)

Then...

v2.0
Do a mod merger (maybe using a system similar to Meph's search/replace concept; mods that don't merge though, would have to be figured out somehow so the proper flags wouldn't conflict)

Then maybe an

v3.0
advanced object merger.

We're waiting for a mansion when all we need is a shack atm.

We literally have ALL the tools to create a batch based mod system.  User clicks on a batch file or linux shell script, then gets a bunch of text prompts asking what mod he wants to load.  Then the mod is applied, just before the game is started from the gui or batch file.

We could EVEN HOST A GITHUB repository of all the mods so a user can go and download them

Hell, we could use github to create our PATCH files for us.

BTW, this is just a proof of concept, but it could easily be done rather than having a user have to hunt mods down and parse and merge.

However, in the meantime, I'd propose just including a few mods, say 5 to 10 in an initial release, and work on a repo where users can upload their mods.  Hell, the theory is easy enough for anyone to implements.  The batch file could simply list a subfolder's dir contents where the mods are stored.... and parse from their.  The batch file would automagically parse and patch the mod compared to a base vanilla version, which is entirely possible using diff3.

I was going to suggest something similar for a repository, with a directory of download locations hosted on DFFD or something, with a fallback to its last-downloaded copy on the local machine.  There is a DFHack scripts collection that uses some wiki tool to make a human-readable directory, but that might be overkill.  All we need is some XML that has the mod title, version number, author, two-sentence blurb, and URL.

The mod tool may end up using a non-standard diff format, but an important design goal is to let the launcher accept raws as input (either making its own diffs, or merging things on the fly) so that a mod doesn't need to be specifically prepared to work with the launcher.  If we want launcher-prepped mods in a special format, we might as well go with Rubble because it already exists and it's kinda awesome.

When the user pastes in a URL for something not in the directory, it should suck the mod in (whether in raw or diff format) and treat it like anything else.

[PeridexisErrant]

I've already written a simple console-based version in Python (3.4).  It even goes through and creates a unified diff file for each file as it goes, and it would be easy to add a section that called the unix patch utility each time.  Unfortunately that doesn't come on windows, and I don't want to pursue something that might never work better.  Here it is:

https://github.com/PeridexisErrant/Py-Mod-Loader



Rubble is indeed awesome, and does the whole "special input == awesome output" thing much better than we're able to.  A far-future stretch goal might be an option to switch over to rubble... after we put a roof on our modest house. 

Hosting is, by design, wherever mods are found.  I imagine that this will come with a decent set to start with though!

[thistleknot]

Note:

The fear of a tool "never" getting better should never stop one from implementing it.  Things can be proof of concept until we get better features implemented.  A python based gui could be in order next.

I'm not 100% sure how your python script works.

I would imagine the concept would do 1 of two things.

A 'ModTool' (whatever name) which scan two folders

One for dropped in raw's, say

\ImportedMods

and one for already created patchffiles and maybe a matching zip of binary files that need to be included
\PatchFiles

Inside \ImportedMods\ would be:
  .\Base\ (Based on Vanilla)

Inside \PatchFiles\ would be
  at least 5 patch files for 5 different mods.

A User can drop a mod subfolder into \ImportedMods\ as
  .\<NameOfMod>\
subfolders that would [need to] be copied in would be dataand raw.

When the script starts, it lists the choices for what to load based on the contents of these two subfolders.  So... when running, a user would choose 1 of two options:

1. Apply patch (Load Mod)
2. Process mod[ s] (Create Patch[es])

Option1:

Fairly straightforward.  Lists contents of \Patchfiles\, lets a user pick a patchfile & applys it against \ImportedRaws\Base\ and outputs the results into a [different] folder which is basically the contents of Dwarf Fortress Vanilla but with empty subfolders for data and raw, which are to be populated by the patch output.

Option2:

a. Flattens all \ImportedMods\*\Raw\Objects\*.txt

b. and we do a diff comparison between (including all subfolders, missing files between base and <NameOfMod> have blank base placeholders created)

  \ImportedMods\Base\data\*.*
  \ImportedMods\Base\raw\*.*
and
  \ImportedMods\<NameOfMod>\data\*.*
  \ImportedMods\<NameOfMod>\raw\*.*

either as a single <NameOfMod>.patch patch file or <NameOfMod>data.patch and <NameOfMod>raw.patch set of patch files.  Also, the patch file(s) ignore binary files; instead, binary files that differ are saved into a matching zip file.

and dump the patch file (with the matching zip of binary files that were different) into

\PatchFiles\

The one exception to merges for atm should be "tileset" patching on top of mods.  Siimilar to how the linux java based lnp works.  However, we could simply do a diff3 comparison using a common ancestor.  There hopefully won't be any conflicts, and a matching diff3 output file can be created using the concept.

The beauty of this system, is it leaves the complicated [possibly broken] merging of mods up to mod mergers who know how to hand implement merged mods.  As we know, it's not a pretty process.  That way when a mod is intelligently merged, we have a format for them to create a patch file from it.  That patch file can BE PASSED AROUND

[Dirst]

So thistleknot, if I read that correctly, you'd like the launcher to apply one and only one patch?

I'd prefer PE's idea of specifying the order and throwing warnings about potential conflicts, probably using some version of diff3.  That way someone who wants to use Modest Accelerated Mod and Mushroom Kingdom doesn't need to hunt around for a patch that pre-mixes the two.

That said, I have three future-proofing suggestions:

1. Treat graphics packs like any other mod.  In fact maybe define arbitrary roots for the mod's folders so that things like Stonesense and Soundsense content can come along for the ride.

2. Allow the optional manifest file to silently okay some apparent conflicts and explicitly reject others.  These should be version-number specific.  This allows a mod to have sub-mods that depend on the core.

3. Try to encourage PE's idea of modular mods.  This will reduce (though not eliminate) collisions.  That entity_default.txt file is going to be prodded a lot.

[thistleknot]

I'm merely suggesting if we're having a hold up based on what is ready vs not ready.  Currently, no one is 100% onboard with the way diff3 does merging.  It can drop lines that are meant to be merged together.

So, since that requires more advanced merging techniques, I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod.

I do see one issue with tilesets, and that's when we have mods that bring in new creatures, plants, etc... we can't rely on a universal tileset applier for all mods... (for the most part it might work though).

As to modularity:

I think if modders keep their changes to new_files vs changing vanilla files.  It would help with merging mods...  However, a lot of mods mod objects in vanilla files... so unless the object was removed and put into an external file to be modded...

[Dirst]

I'm merely suggesting if we're having a hold up based on what is ready vs not ready.  Currently, no one is 100% onboard with the way diff3 does merging.  It can drop lines that are meant to be merged together.

So, since that requires more advanced merging techniques, I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod.

I do see one issue with tilesets, and that's when we have mods that bring in new creatures, plants, etc... we can't rely on a universal tileset applier for all mods... (for the most part it might work though).

As to modularity:

I think if modders keep their changes to new_files vs changing vanilla files.  It would help with merging mods...  However, a lot of mods mod objects in vanilla files... so unless the object was removed and put into an external file to be modded...

Argh, didn't mean to imply we shouldn't go forward with a one-mod-only version to start.  I just don't see that as a viable end-goal.

If two mods try to change a single line (such as the grazing stat for a specific creature), it ought to end up with a last-in-wins scenario or at least an error.  Dropping lines would be a bug in diff3 and they should probably hear about it.

Some mods add stuff, and those should be in creature_mymod.txt and inorganic_mymod.txt and so on.  Some mods explicitly fiddle with vanilla stuff (such as the thirty-seven or so variations on hard farming), and those should change the tags they need to change without trying to tidy up the vanilla stuff around them.  It's unlikely you'd get two hard farming mods to co-exist, but you should at least be able to get a hard farming mod to work with an arbitrary graphics pack.

If a modder believes that his or her change to a vanilla object needs to be mutually exclusive of any other modding, then the mod should replace the vanilla entry with a comment and recreate the object in a new file.  That will give the patch loader a tummy ache and prevent the incompatible mixing.

[thistleknot]

I'm merely suggesting if we're having a hold up based on what is ready vs not ready.  Currently, no one is 100% onboard with the way diff3 does merging.  It can drop lines that are meant to be merged together.

So, since that requires more advanced merging techniques, I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod.

I do see one issue with tilesets, and that's when we have mods that bring in new creatures, plants, etc... we can't rely on a universal tileset applier for all mods... (for the most part it might work though).

As to modularity:

I think if modders keep their changes to new_files vs changing vanilla files.  It would help with merging mods...  However, a lot of mods mod objects in vanilla files... so unless the object was removed and put into an external file to be modded...

Argh, didn't mean to imply we shouldn't go forward with a one-mod-only version to start.  I just don't see that as a viable end-goal.

I didn't intend it to be an end-goal, but rather a start

I know I said,

"I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod."

but... I didn't mean that be our "final" product; but, rather, due to issues with diff3 (for example) and other parsing methods that are preventing us from building a merger from the start... just do a simple patching system at first, then build from that.

Btw your idea of using blank placeholders for heavy diff files is similar in concept to if a file does not exist in vanilla.  How would one code that, like, right now.  Is that possible without having to implement ###comments... how would that be handled/flagged appropriately?

I think another cause for issue is the fact that some of us have ideas in one language (primarily myself in batch scripting and command lines) and others using python.  I'm all for python; however, the py is weak in this one   It does have the added affect that while we discuss the possible "end-case" scenario's on the thread... anyone one of us could have completed a simple batch solution in our own right that could have served as a base 1.  I do think if python is more versatile, to go that route vs shell scripting.  Just ensuring that the end client dependencies are met when running python (without conflict) are ensured, then go for it (DFHack does it somehow).

[Dirst]

Argh, didn't mean to imply we shouldn't go forward with a one-mod-only version to start.  I just don't see that as a viable end-goal.

I didn't intend it to be an end-goal, but rather a start

Okay, we got that miscommunication out of the way.  On to business.

Btw your idea of using blank placeholders for heavy diff files is similar in concept to if a file does not exist in vanilla.  How would one code that, like, right now.  Is that possible without having to implement ###comments... how would that be handled/flagged appropriately?

Suppose someone heavily modifies the Elven entity in their mod, like adding a dozen nobles and stylings while tweaking just about everything else.  The odds of that successfully merging with even a modest change in another mod (such as progress triggers) is close to nil.  So the idea would be for the modded file to have a comment in its entity_default.txt along the lines of

Note: The ENTITY:FOREST object has been moved to entity_woodstock.txt by Woodstock Mod v1.08.

and delete every line of the original entity.  This will cause a conflict with any other attempt to modify the Elves' entity (except someone trying to delete it completely).  It doesn't need any special logic in the launcher, just some advice for modders who want their stuff to work better with the launcher.  And it doesn't prevent someone else from modifying a different vanilla entity, either.

I think another cause for issue is the fact that some of us have ideas in one language (primarily myself in batch scripting and command lines) and others using python.  I'm all for python; however, the py is weak in this one   It does have the added affect that while we discuss the possible "end-case" scenario's on the thread... anyone one of us could have completed a simple batch solution in our own right that could have served as a base 1.  I do think if python is more versatile, to go that route vs shell scripting.  Just ensuring that the end client dependencies are met when running python (without conflict) are ensured, then go for it (DFHack does it somehow).

Shell scripting is going to be difficult or impossible to keep cross-platform, so I would lean toward Python.  Or at least I would if I knew how to script in Python.  As it is I just point over to where the Python lives and encourage others to go over there and build a shining city on the hill.

[PeridexisErrant]

Note:

The fear of a tool "never" getting better should never stop one from implementing it.  Things can be proof of concept until we get better features implemented.  A python based gui could be in order next.

I'm not 100% sure how your python script works.

I would imagine the concept would do 1 of two things.

A 'ModTool' (whatever name) which scan two folders

One for dropped in raw's, say

\ImportedMods

and one for already created patchffiles and maybe a matching zip of binary files that need to be included
\PatchFiles

<snip>

The one exception to merges for atm should be "tileset" patching on top of mods.  Similar to how the linux java based lnp works.  However, we could simply do a diff3 comparison using a common ancestor.  There hopefully won't be any conflicts, and a matching diff3 output file can be created using the concept.

The beauty of this system, is it leaves the complicated [possibly broken] merging of mods up to mod mergers who know how to hand implement merged mods.  As we know, it's not a pretty process.  That way when a mod is intelligently merged, we have a format for them to create a patch file from it.  That patch file can BE PASSED AROUND

I object to the idea of distributing the mods as diffs, because the whole point of this thread is that we have a common input format that any tool or version of a tool can use, which won't change with improved logic.  This means that any patch files should be derived each time the tool is run from the basic format.  Graphics are currently beyond scope - we can handle those once the basic thing is working. 

How my script works:

• Process any mods in the LNP/Mods/ folder into the format defined in this thread.  Delete the temp folder used in the last run if it's still there.
• Get an ordered list of mods to load from the user.
• Copy the vanilla raws to the temp folder
• For each mod in the list of mods to load:
• For each file in that mod:
• If that file already exists in the temp folder, create a unified diff (but I can't work out how to apply it with Python)
• Otherwise copy that file to the folder
• Cleanup and verify (not yet implemented)
• Done!  User now has a merged set of raws.