Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Abadrausar]

...You do realize the dungeon master doesn't exist anymore, and we've had a working animal taming system for over two years now, right?

Yes Palu, I know... We have now even more than that with IndigoFenix druidismhttp://dffd.wimbli.com/file.php?id=8583
But I have been tinkering with the problem of how correctly integrating multiple mods in Dwarf Fortress. As you know there are a whole lot of people that have researched or are researching different ways of doing this, and some have implemented partial solutions:

• http://www.bay12forums.com/smf/index.php?topic=140853.0
• http://www.bay12forums.com/smf/index.php?topic=125997.0
• http://www.bay12forums.com/smf/index.php?topic=1422 95.0
• http://www.bay12forums.com/smf/index.php?topic=102316.0

I think that Rubble deserve more attention as a promising way to takle this problem some day, so I am trying to teach myself how to use rubble features to do the same thing that another utility: noexotic.exe, not for his intrisecal value, it is more as a learning experience. So I would be grateful if someone that knows rubble better than me were kind enought to offer some feedback about the rexscript that does not work. 

[PeridexisErrant]

But I have been tinkering with the problem of how correctly integrating multiple mods in Dwarf Fortress. As you know there are a whole lot of people that have researched or are researching different ways of doing this, and some have implemented partial solutions:

• http://www.bay12forums.com/smf/index.php?topic=140853
• http://www.bay12forums.com/smf/index.php?topic=125997
• http://www.bay12forums.com/smf/index.php?topic=142295
• http://www.bay12forums.com/smf/index.php?topic=102316

This comes to basically three strategies:

1.  Just copy files over the top  (Manilla launcher).  Easy for coders.  Modders provide a raw folder with only changed files.  Users just select desired load order and get feedback on validity.  Often can't merge mods which can work together.

2.  Line-by-line merging, and reject a mod that overwrites a previous change (Me, PyLNP soon).  Moderate difficulty for coders.  Modders can provide a changes-only raw folder as above, or users can drop in any modded install of DF and it'll format it.  Users select desired load order and get live feedback.  Can almost always merge simple mods, but may create problems with dependencies or where files are added. 

3.  Comprehension and creation of valid raws (Blast, Rubble).  Hard for coders!  Modders need to use special format and/or write scripts.  Users need to understand modding to a greater extent.  Capable of literal modding magic once thought impossible.

That option two now exists makes it (imo) better than option one, but whether you prefer that or option three depends largely on the circumstances.  Rubble is clearly better for the creation of mod packs, but I think there's a place for something easier for non-technical users as well.  So, uh, I hope people use rubble to create mods for the new Mod Starter Pack 

[milo christiansen]

First to answer Nopenope's problem: The dctech package is included with Rubble, and dctech/patch *should* be there, check "other/src/dctech/patch".

Abadrausar: It is much harder to track your problem down without being able to see line numbers but I think I can identify it: namely

Code: [Select]

#[tag replace = (str:add "[MOUNT]")]
[tag replace = (str:add "[MOUNT" [tag 1] "]")]
[tag 1] is not valid, as MOUNT_EXOTIC does not have parameters.
This should work for that:

Code: [Select]

[tag replace = "[MOUNT]"]
The reason that the commented out line failed you is that str:add requires more than one parameter, so just leave that part out and you are golden.

On a related note: How do you find Rex? (the scripting language) Is it easy to use and understand? I ask because it is (asside from Rubble) my big project right now, and knowing how others see it would be very helpful.

@PeridexisErrant: How do users need to understand modding? You just select a mod to install and Rubble takes care of the details. Modders on the other hand DO need to do some more work, but I find that after the initial learning curve you actually need to do LESS work (templates save a lot of typing, and scripts automate boring, repetitive tasks).

==================================================

4.7 will be up shortly, I am still behind on the version curve, but it sounds like 40.12 doesn't make many (if any) changes to the raws.

The big deal this version is that all the DFHack stuff is finally working!
I did finds some possible bugs in DFHack, but I was able to avoid them, so all is good.

I, of course, did some other stuff as well, mostly bugfixes and minor additions.

[PeridexisErrant]

@PeridexisErrant: How do users need to understand modding? You just select a mod to install and Rubble takes care of the details. Modders on the other hand DO need to do some more work, but I find that after the initial learning curve you actually need to do LESS work (templates save a lot of typing, and scripts automate boring, repetitive tasks).

There are a couple of things that are going to be important to users.  One is whether they decide to use the tool, and to beginners Rubble looks pretty intimidating; eg "once you learn a new scripting language it's easy!" - I don't doubt this for a second, but a lot of people won't try.  The documentation also looks pretty intense and assumes you know what you're doing.  Whether it's actually easy to use or not only becomes a factor after they try, at which point it had better be!

Another is content.  How much is included by default?  How much more can be found in the appropriate format?  How hard is it for a user to translate a mod into the format?  To add a properly formatted mod?  Rubble is miles ahead on default content, and you can add a pre-formatted mod to any tool just by dropping the folder in the right place.  The problem, in my mind, is the input format - availability, generality, and ease of generation. 

My less powerful system has a couple of advantages on the input side - in fact, most early development was entirely focussed on the input format.  It can be automatically created from an installed mod (and I plan to add ways to transform other common distribution formats).  It's meaningful in the absence of a merge tool - you can just copy the mod across a vanilla DF install - and also more likely to be cross-compatible with other tools, which will hopefully help build a standard.  In short, it's designed for interoperability at the expense of excellence.  Frankly I also underestimated the sheer variety and obtuseness of the formats in which people distribute mods - I'm hoping leveraging the starter pack can help fix that. 

By contrast Rubble is more demanding to get into, though once you're in it is a lot better.  The format is super-specific to Rubble, and that means it's less likely to get the diversity or adoption rate that would drive content.  It's far more useful to people who do a lot of modding, maintain their own mod collections, publish combinatorial sets of mods, or generally want to do something "advanced"... but the barriers to entry are that much higher, and it shows. 

The best of all possible scenarios would be if Rubble could be extended to take mods and decompose them to Rubble inputs.  It could be tricky, but then again I recently read about someone who actually coded a program to play regex golf for him!  Then just steal my more basic logic, hook it all togther with the LNP, and mods happily ever after!  Until then, different tools for different uses I guess   

[Dirst]

The best of all possible scenarios would be if Rubble could be extended to take mods and decompose them to Rubble inputs.  It could be tricky, but then again I recently read about someone who actually coded a program to play regex golf for him!  Then just steal my more basic logic, hook it all togther with the LNP, and mods happily ever after!  Until then, different tools for different uses I guess   

An intermediate step would be to hook the Option 2 launcher into the Option 3 tool(s) on the back end.  So the user lines up a set of mods in the front end, and the Launcher notices that one of them is in Rubble format.  When the Launcher reaches that mod, it uses an API or command line calls to get Rubble to do its thing on that file. That detection can come from a manifest file or some explicit version flagging in the first line of the file.

The question then becomes: Does the Level 3 tool simply down-convert the mod into something the Level 2 tool can merge, or does the Level 3 tool merge its mod into the temporary build?  Remember, the Level 3 mod isn't necessarily first or last in order.  Ideally, both options would exist via API or command line flags, with non-default behavior specified by the manifest or an "Advanced" setting by the user.

[PeridexisErrant]

The best of all possible scenarios would be if Rubble could be extended to take mods and decompose them to Rubble inputs.  It could be tricky, but then again I recently read about someone who actually coded a program to play regex golf for him!  Then just steal my more basic logic, hook it all togther with the LNP, and mods happily ever after!  Until then, different tools for different uses I guess   

An intermediate step would be to hook the Option 2 launcher into the Option 3 tool(s) on the back end.  So the user lines up a set of mods in the front end, and the Launcher notices that one of them is in Rubble format.  When the Launcher reaches that mod, it uses an API or command line calls to get Rubble to do its thing on that file. That detection can come from a manifest file or some explicit version flagging in the first line of the file.

The question then becomes: Does the Level 3 tool simply down-convert the mod into something the Level 2 tool can merge, or does the Level 3 tool merge its mod into the temporary build?  Remember, the Level 3 mod isn't necessarily first or last in order.  Ideally, both options would exist via API or command line flags, with non-default behavior specified by the manifest or an "Advanced" setting by the user.

Yep.  In the medium to long term I'm keen to hook in Rubble as an alternative back-end.  Using the (forthcoming) manifest for this would be a good idea.

The challenge to combining them is that Rubble requires some hooks to build off.  It's almost certainly going to be easier to build the Rubble components, post-process them to remove extra tags and so on which might cause merge failure, and then use Option 2 merging.  The alternative would be to have Rubble comprehend the raws without hooks, which is obviously pretty difficult. 

At some point I want a raw-validity-checker, so we can just keep trying to combine things in different ways until it works 

[milo christiansen]

About Rubble and users:
You make some good points, some of which I had not thought about...

For users the experience is supposed to be: Just extract Rubble to "<df>/rubble", launch an interface, select your addons, and go!
I suppose I should make it more clear that the documentation is strictly "by a modder for modders" and should be ignored by most users, but that probably won't help anything.

One thing I could do is use my raw parser to decompose a mod and convert it as much as possible into something that would play nice with the Rubble system, the problem is that such a thing would have to be an external tool, trying to build such capabilities into Rubble would be doomed to failure. The main issue is that Rubble STRONGLY discourages modifying existing files, for nearly every time you would modify an existing item, Rubble provides a template, so converting raws would require a lot of comparing vanilla and mod raws, one object at a time looking for changes and the deciding how to merge those changes, for some changes it would be very difficult to implement a generic system that would work in all cases. Anyway it would be far too much work for right now, maybe later...

Yep.  In the medium to long term I'm keen to hook in Rubble as an alternative back-end.

Rubble-as-service? I could do that...
The problem is that as things stand Rubble requires on ADDON_HOOKS and SHARED_OBJECT (specifically SHARED_MATERIAL_TEMPLATE and SHARED_INORGANIC are the most common). Without those two templates (and their many supports) Rubble is crippled. It would be possible to hook these templates' output, but the problem is "how do you want to communicate to Rubble?"

The easiest way (for me) would be via a limited http server, but other types of servers (including several types of RPC) would be possible. Sadly it would not be possible for me to provide a DLL for low level access, as go (the language I used for Rubble) cannot create DLLs.
Command line would work as well, but you would have to parse output from stdout, which can be a real pain in the rear...

In short, to use Rubble as a secondary backend you would have to provide an interface that would act as a backend for most of the standard templates, which would be quite the job (it would also require me making a low enough level interface, but if the need is there I will do it).
Mostly you would get lists of tags to add/remove/modify to specific raw objects, reaction/workshop names to add to specific entities, etc (and don't forget that you cannot actually do anything until ALL the raws are parsed, because other later templates may undo earlier actions!).

=========================================================
Rubble 5.0 will be up shortly!

5.0? Why not 4.8?
Well I rewrote large chunks of the internals and turned Ruble into a library with several thin interfaces on top. The first interface is the good old CLI, but the second is a brand new web UI!
The web UI is a little ugly (I suck at HTML) but it works fine and is MUCH easier to use than the CLI, and has far more features compared to the old GUI.

Also added is the "addon.meta" file for handling the new addon meta-data. This data includes dependencies, descriptions, and a special flag for libraries. Most of the meta-data is not handled directly by Rubble, but what Rubble doesn't use is used by the web UI
For example there is a page in the web UI that lists all your installed addons beside a one line description, click on the addon name and it takes you to a page with details about the addon, including a full description, a list of it's dependencies, and a list of all it's files (which can be viewed by clicking on the file name).
addon.meta is the core of the new automatic dependency management system Rubble uses (to old way still works, and will not be removed).

[PeridexisErrant]

One thing I could do is use my raw parser to decompose a mod and convert it as much as possible into something that would play nice with the Rubble system, the problem is that such a thing would have to be an external tool, trying to build such capabilities into Rubble would be doomed to failure. The main issue is that Rubble STRONGLY discourages modifying existing files, for nearly every time you would modify an existing item, Rubble provides a template, so converting raws would require a lot of comparing vanilla and mod raws, one object at a time looking for changes and the deciding how to merge those changes, for some changes it would be very difficult to implement a generic system that would work in all cases. Anyway it would be far too much work for right now, maybe later...

Yep.  In the medium to long term I'm keen to hook in Rubble as an alternative back-end.

Rubble-as-service? I could do that...
The problem is that as things stand Rubble requires on ADDON_HOOKS and SHARED_OBJECT (specifically SHARED_MATERIAL_TEMPLATE and SHARED_INORGANIC are the most common). Without those two templates (and their many supports) Rubble is crippled. It would be possible to hook these templates' output, but the problem is "how do you want to communicate to Rubble?"

The easiest way (for me) would be via a limited http server, but other types of servers (including several types of RPC) would be possible. Sadly it would not be possible for me to provide a DLL for low level access, as go (the language I used for Rubble) cannot create DLLs.
Command line would work as well, but you would have to parse output from stdout, which can be a real pain in the rear...

In short, to use Rubble as a secondary backend you would have to provide an interface that would act as a backend for most of the standard templates, which would be quite the job (it would also require me making a low enough level interface, but if the need is there I will do it).
Mostly you would get lists of tags to add/remove/modify to specific raw objects, reaction/workshop names to add to specific entities, etc (and don't forget that you cannot actually do anything until ALL the raws are parsed, because other later templates may undo earlier actions!).

Yeah.  Parsing is hard, and the better you want the output to be the more expensive it gets.  Good choice to leave it for later!

I obviously haven't given much thought at all (beyond "that would be nice") to integrating Rubble.  It would probably come down to popping up a rubble interface to build a mod, then taking that as the first thing to merge and 'cleaning' it of the hooks to avoid causing diff/patch issues. My proposal:  we leave this at least until the loader is finished (next year probably).  It could also be a direct alternative instead of integrated, which would be easier for everyone...  so don't bother doing anything for me for a while!

[thistleknot]

ahem... static patch system...

update...

of course...
what does that accomplish...

DEFACTO MODS

[milo christiansen]

I think I am over thinking things...

The easy way to integrate Rubble would be to use Rubble to generate a basic set of raws (including any addons the user wants) and then apply any other mods to that base.
It may be hard to apply patches to some of the files, but a bit of preprocessing of the patches and some "markers" inserted into the base (to specify, for example, where to add reaction permissions or whatever) should take care of that just fine.
Any system I can come up with for allowing Rubble to do partial generation ends up failing upon the rocks of being far too hard on the person trying to integrate it (you would need to interact with the Rex state at a low level, and an understanding of the workings of the individual templates would be required)

Another thing that may help is that Rubble can read normal raws just fine, it just doesn't do anything to them (except adding file headers), so you can just pass any raws you want right on through if something is easy to add before generation rather than after...

================================================

Rubble 5.1 will be up shortly, no huge changes, but several little bug fixes and a version update.

[Abadrausar]

...
On a related note: How do you find Rex? (the scripting language) Is it easy to use and understand? I ask because it is (asside from Rubble) my big project right now, and knowing how others see it would be very helpful.
...

I think you are doing a very nice job modelling the complexity of moddding Dwarf Fortress (DF) because not all complexity can be cached without losing key parts of a system.
Rex as I see it is a scheme-like kind of language (where The most widely implemented standard is R5RS) Rex is adapted from the beginning to understand and modify another language, the declarative raws of DF.

Between those two the original Rubble language, much more ad hoc and limited but more easy to understand, it is just ok if it solves your current problem

By now I am adapting use cases of the rubble addons to my owns modding needs, so I learn little by little. In this sense I dont have in this moment an opinion closed over the rubble system, I only enjoy trying new things with it.

I have tried every single addon in rubble and two of them seems to have errors in the version 5.1, I leave you the logs of the errors in the generation:
1)Research

Spoiler (click to show/hide)

 reaction_user_research.txt
Abort: Script Error: "foreach": Attempt to range over non-Indexable. Near: libs_research.pre.rex|L:119|C:2

2) dfhack_transform

Spoiler (click to show/hide)

  reaction_user_dwarf_castes_dfhack_transform.txt
panic: Lexer template depth less than 0 (Unmatched curly brackets) [recovered]
   panic: Lexer template depth less than 0 (Unmatched curly brackets)

goroutine 1 [running]:
runtime.panic(0x51ff00, 0x12db1e40)
   C:/go/src/pkg/runtime/panic.c:266 +0xa6
rubble.trapAbort(0x34255c)
   C:/Docs/Projects/Go/src/rubble/state.go:105 +0xbb
runtime.panic(0x51ff00, 0x12db1e40)
   C:/go/src/pkg/runtime/panic.c:248 +0xe8
rubble.(*Lexer).Advance(0x13da1c80)
   C:/Docs/Projects/Go/src/rubble/lexer.go:95 +0xfc
rubble.(*State).ParseFile(0x121c0700, 0x1417d000, 0x724, 0x19000, 0x5, ...)
   C:/Docs/Projects/Go/src/rubble/parse.go:41 +0x105
rubble.(*State).Parse(0x121c0700, 0x0, 0x0)
   C:/Docs/Projects/Go/src/rubble/state.go:596 +0x162d
rubble.(*State).Run(0x121c0700, 0x121b0360, 0x1, 0x1, 0x0, ...)
   C:/Docs/Projects/Go/src/rubble/state.go:333 +0xa1
main.main()
   C:/Docs/Projects/Go/src/rubble/interface/cli/main.go:228 +0x1633

Maybe a check option in the rubble command line could have some use, This option could iterate over all not incompatible adddons and loggin the errors.

[Dirst]

Spoiler (click to show/hide)

  reaction_user_dwarf_castes_dfhack_transform.txt
panic: Lexer template depth less than 0 (Unmatched curly brackets) [recovered]
   panic: Lexer template depth less than 0 (Unmatched curly brackets)

goroutine 1 [running]:
runtime.panic(0x51ff00, 0x12db1e40)
   C:/go/src/pkg/runtime/panic.c:266 +0xa6
rubble.trapAbort(0x34255c)
   C:/Docs/Projects/Go/src/rubble/state.go:105 +0xbb
runtime.panic(0x51ff00, 0x12db1e40)
   C:/go/src/pkg/runtime/panic.c:248 +0xe8
rubble.(*Lexer).Advance(0x13da1c80)
   C:/Docs/Projects/Go/src/rubble/lexer.go:95 +0xfc
rubble.(*State).ParseFile(0x121c0700, 0x1417d000, 0x724, 0x19000, 0x5, ...)
   C:/Docs/Projects/Go/src/rubble/parse.go:41 +0x105
rubble.(*State).Parse(0x121c0700, 0x0, 0x0)
   C:/Docs/Projects/Go/src/rubble/state.go:596 +0x162d
rubble.(*State).Run(0x121c0700, 0x121b0360, 0x1, 0x1, 0x0, ...)
   C:/Docs/Projects/Go/src/rubble/state.go:333 +0xa1
main.main()
   C:/Docs/Projects/Go/src/rubble/interface/cli/main.go:228 +0x1633

I'm assuming that was a real panic error?!

That's one thing I always liked about *nix systems, it will calmly inform you when it's time to panic.

[Taverius]

Mmm. I remember trying this back in v1.8, and hitting a wall face-first at speed.

So I went back to modding Kerbal Space Program.

Tried it again, hit the same wall.

Do I really have to use Rex to edit a single entity definition and replace one its tags? Because if so, you need a DSL for that kind of usage, and none of the (fairly impenetrable) documentation suggests different.

[milo christiansen]

@Taverius: Yes you will probably need to use Rex for that, depending on what kind of tag you are trying to replace of course (some things have special templates)
Entities are one of these things that do not have SHARED_OBJECT support, so the standard SHARED_OBJECT_KILL_TAG won't work there, sorry.

@Abadrausar: The first error is probably minor, once again I forgot to test something, *sigh*. The second error on the other hand, that looks bad. Very bad.
Last time I used that addon it worked fine and I don't think I made any changes since then, so the problem is almost certainly in the Rubble program itself.
(BTW: error handling uses go's panic/defer/recover system internally to allow escaping from deeply nested function calls, in this case the panic was not properly recovered, so a simple error looks WAY worse than it should)

That check idea is starting to look really good...

[milo christiansen]

I am feeling really stupid. I spent a great deal of time working on Rubble over the last few days, fixing bugs (only one major) and adding new features, I even stayed up late last night to make sure everything would be ready, and it was all ruined by me forgetting to load the file onto my flash drive before setting out to the library.

So 5.2 will be delayed by about a week.

The most important upcoming features are:

• Web Load addons, create a file containing a URL to a zip file and Rubble will download it as needed (but only if the server's copy is different from the local copy)! Automatically updating addon packs will soon be possible!
• Better error reporting, several redundant error handling types are replaced with a single more flexible version and the whole system is more accurate.
• The "Libs/Research" and "User/Research" addons will actually work!
• Rex scripting will have an expanded syntax for command parameter handling.
• And anything else I add before I get to a library again