Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Abadrausar]

...

Apology accepted Milo.

Even if taking care of the feelings of the people about things is important to have a strong and united community, in reality what were worrying me the most were the feeling of lost opportunity...

What I mean is that any successful software project needs a bug database as complete and detailed as posible, because that is like the intelligence reports for an army, if you have it good, you can take your objetives, you can save your people and you can assign your resources effectively, if you have it bad, surely your army is about to be overwhelmed because without good infos the army is blind.

In this sense you can even measure the success of a software by the diversity, quality and quantity of their intelligence reports (their bug databases), if anyone doubt it only has to consult the bug databases of projects like Dwarf Fortress, DFhack or Linux.

But reporting a bug requires more work than grumbling mildly and deciding than a tool is not helping us, it is sad to say that, but most of us are really lazy, have little time or both.

That means that the process of bug reporting, so valuable for us, should be done as much as possible without drama ni confrontation, in an aseptic way because if we fail in that, people will report scarcely or not at all, and that is against our interest.

Only my opinion, but open to other views...

I just have downloaded First Landing 1.7 and Rubble 7.7, I will compare them with their precedent versions with gitextension and if I detect some problem the relevant info will be at your disposal.

[milo christiansen]

There will never be a bug tracker so long as I do not have internet,  there will also not be a public repository, fast turnaround times for bugfixes, and a dozen other things that would be really nice to have.

Without easy access to both the internet and my development/game machine, and with turnaround time measured in days or weeks between me finding out about a bug and me being able to do something about it, bug reporting and fixing will both remain frustrating.

But, I may get internet this summer (I hope), and that would help a lot (faster communications loop).

[Dirst]

Since it's just a plain text file, is there a way to get the module to spit out a .webload file in the right place at raw-generation time?

Once the system is stable (that is, functions when the module itself is not yet installed), I can add instructions on making a .webload into my mod's OP.

[Abadrausar]

Milo I have tried your rubble conversion addon SO Insert and it seems to work, at least at a basic level as you say, luckily this is what is needed most of the time.
Proposal to streamline the process of converting Dwarf Fortress raws to the rubble system:
RFC: Adding a new File Extension and Tag .DF with every file of a mod containing unmodified Dwarf Fortress objects
In a stage anterior to any other in rubble, any files with extension .DF will be feeded to the SO Insert subsystem that will transform them into the correspondent  .txt files with the Dwarf Fortress objects already rubblized, following that, will begin the normal rubble process as we know it now.

Bonus points if most of this registration process can be done in lua lands instead of in go lands, mostly because I would like to learn to contribute myself some way to declare a file extension as being .world generation parameters and thus being effectively added to the end of the data/init/world_gen.txt file or to data/init/embark_profiles.txt for the extension .embark, or many other as speech...

I could try to do those little additions with lua, but hardly with goland...
There are between us more lua noobs, like me for example, than go noobs.

[milo christiansen]

If you want to handle more extensions (via Lua instead of adding native code to handle them) it is certainly possible. Currently you will have to parse the file name yourself (which, considering the state of the Lua string library, is a real pain) and decide how you want to to handle each file individually. Depending on the file's existing last part extension it may be possible to use the file action API (look at the script that applies extra file tags to raw files for a great example), but if the file type you want to handle does not have any tags then you will have to walk the addons and their files yourself.

Once you find a file of interest you have a choice, handle it now, or tag it and handle it later. I personally recommend tagging files in a load script, then writing them out in a post script with a sequence of "zzz" (as this way the vast majority of possible failures will be past).

Obviously this is cumbersome, but it would be impossible for Lua scripts to add rules to the file tagger, as this is run at load time, long before it is possible to run scripts.

Since it's just a plain text file, is there a way to get the module to spit out a .webload file in the right place at raw-generation time?

Nope. Rubble has no way to know where you will upload the addon pack or where you downloaded the pack from. Oh, wait. You want it to install a .webload so that users will have automatic update without needing to install a .webload too? In that case yes, but you will have to use a script to write a file to "addons:dir:" and it won't be guaranteed to work properly (what if a user installs your addon under a non-default file name?).

I am currently working on a new version of AXIS VFS that will allow a more flexible addon loader, in particular I plan to change the way Rubble stores loaded addons so that addons are loaded once at startup instead of being loaded fresh every time a new State is created. Obviously this will require the ability to add more addons to the database at any time prior to generation. Also obvious is that it will be possible to add an automatic update feature that does not depend on the existing web load system (which is more useful for initial installation).

The new AXIS and loader will probably not hit soon, but when it does it will bring major changes to addon loading. This (better addon loading) will probably be the main feature of "Rubble 8". I do have to stress that I won't be making such a large jump for a while...


New version! 7.7.1 brings a minor bug fix and basic support for "spoilers". Spoiler support is very basic, but it works. For details see the change log (and there is an example in Rubble Basics, read it in the web UI to see it in action).

[Dirst]

Thanks Milo.

What I was looking for in generating a .webload was something like (in completely made-up pseudocode)

IF NOT EXISTS("addons:dir:The Earth Strikes Back!.webload") THEN
   CREATE WLFILE AS "addons:dir:The Earth Strikes Back!.webload"
   WRITE TO WLFILE "http://dffd.bay12games.com/download.php?id=11912&f=The+Earth+Strikes+Back%21.zip"
   CLOSE WLFILE
END IF

If the user changed the name of the zip, then they'll end up with two copies.  But that's better than a really complicated and brittle round of exception handling.

[Abadrausar]

Report of active BUG in the webload system:

The Rubble changelog for version 7.7.1 says

Spoiler (click to show/hide)

7.7.1 (For DF 42.6) Minor bug fix.
------------------------------------------------------------------------------------------------------------------------

This version fixes one minor bug and adds one "new" feature.

The "new" feature isn't really new, since anyone could already do what it does if they were willing to write a little
bit of JavaScript. This simply makes it slightly easier to do in a consistent manner.

Engine Changes:

* Fixed a bug in the web load system that prevented the downloaded files from being saved to disk.
* Added a bit of CSS and JavaScript to the addon information and documentation pages to support adding "spoilers".
   * You need to use HTML to add the required tags, but markdown allows embedded HTML, so use that.
   * The end of the [Rubble Basics web UI links section][RB1] has an example showing how this works.

Addon and Standard Library Changes:

* None

Other Changes:

* None

What happens in reality is

Spoiler (click to show/hide)

Back to Menu
Exit Rubble
An error has occurred or Rubble has aborted, scroll to the end of the log for details.

Rubble v7.7.1
Runs natively on Windows, Linux, and OSX!
==============================================
Attempting to Read Config File: ./rubble.ini
  Read failed (this is most likely ok)
  Error: open ./rubble.ini: The system cannot find the file specified.
    Using hardcoded defaults.
==============================================
Initializing...
  Creating New State...
  Initializing AXIS VFS...
  Loading Special Global Files...
    Reading DF Init Files...
==============================================
Starting the Default Web Browser...
  Attempting to Open: "http://127.0.0.1:2120/menu"
Starting Server...
UI Transition: "/menu"
==============================================
Initializing...
  Creating New State...
  Initializing AXIS VFS...
  Loading Special Global Files...
    Reading DF Init Files...
==============================================
Loading Addons...
  Mounting Local Zip Files...
    First Landing.zip
Load time:  4.0002ms
Error: While mounting zip file:: zip: not a valid zip file
UI Transition: "/log"
Redirecting: / to main menu.
UI Transition: "/menu"
==============================================
Initializing...
  Creating New State...
  Initializing AXIS VFS...
  Loading Special Global Files...
    Reading DF Init Files...
==============================================
Loading Addons...
  Mounting Local Zip Files...
    First Landing.zip
Load time:  4.0003ms
Error: While mounting zip file:: zip: not a valid zip file
UI Transition: "/log"
UI Transition: "/menu"
==============================================
Initializing...
  Creating New State...
  Initializing AXIS VFS...
  Loading Special Global Files...
    Reading DF Init Files...
==============================================
Loading Addons...
  Mounting Local Zip Files...
  Downloading and Mounting Zip Files from .webload Files...
    First Landing: Downloading.
      Error saving downloaded file: File/Dir Read-only: addons:dir:First Landing.zip
  Loading Addons from Mounted Resources...
  Downloading and Mounting Zip Files from Meta-Data...
  Checking Loaded Data...
==============================================
Activating...
  Clearing Leftover Addon State Information...
  Loading Config Variables...
    No variables specified.
  Generating Active Addon List...
    No addons specified, using default addon list file.
    Loading List File: addons:dir:addonlist.ini
  Activating Addons from Generated List...
  Pruning Library Addons...
  Activating Required Addons from Meta Data...
  Running Loader Scripts...
    aaa_init_scripting.load.lua
  Active Addons:
    Libs/Base
    Libs/Castes
    Base
  Checking for Incompatible Addons from Meta Data...
  Setting Unset Config Variables to Their Defaults...
    Attempting to load rubble:userconfig.ini...
      Could not load rubble:userconfig.ini: File/Dir Not Found: rubble:userconfig.ini
      (This is probably OK.)
    Setting Remaining Variables from Meta-Data...
    (Any variables not listed above were explicitly set elsewhere.)
Load time:  2.1891252s
UI Transition: "/genaddons"
  Loading Presets....
Redirecting: / to main menu.
UI Transition: "/menu"
==============================================
Initializing...
  Creating New State...
  Initializing AXIS VFS...
  Loading Special Global Files...
    Reading DF Init Files...
==============================================
Loading Addons...
  Mounting Local Zip Files...
    First Landing.zip
Load time:  5.0003ms
Error: While mounting zip file:: zip: not a valid zip file
UI Transition: "/log"
Back to Menu
Exit Rubble

If we compare that with the desired behavior

Milo when a .webload exist in the local dir for an addon but no the related .zip file of the addon, could the actual implementation of the .webload system be slightly modified? in order to:
1) Redownload the .zip when the fileserver is accesible.
2) If not, log that the needed file is offline for any reasons and also one message saying that the file will be redownloaded when the file will be accesible again.
3) Never the rubble generation process should fail or block only because one zip is missing when it can be automatically redownloaded later. (with the actual system, if the user did not find the missing .zip he is forced to delete the .webload file  to deblock the rubble generation )

The situation has advanced a little, because now, at least, the downloaded zipped file is created in the local dir, but it has always size 0, at the source of the error, apparently, a problem with the write permissions of the zipped file in the local dir, at least that is what is stated by the error message.
This blocks rubble generation as described in point 3)

In resume, the bug in the webload system is not fixed: Now the downloaded file is created at the local dir, but empty, so the downloaded zipped file is not really saved in the local dir as it should.

I use windows 7 and google chrome as navigator for testing but I have tried also with others (firefox and opera) and the behavior is the same.

[milo christiansen]

Damn... I have no idea why that doesn't work, it did in my (admitedly not very comprehensive) testing. I'll do some more extensive testing across my LAN and see what happens.

Spoiler: Technical Stuff (click to show/hide)

If the WriteAll call fails due to read-only the Create call should as well (but it obviously isn't), this is probably an AXIS error... Generally a read only error is a "soft" limit set when the DataSource is created, but that should not be the case here. I have no idea what is causing it, but it may have been an accidental change while I was working on AXIS 2 after I did the testing. This should not cause any problems with AXIS 1 (as they are separate packages), but I had some AXIS 1 files open for reference, so I may have edited the wrong thing.

If the user changed the name of the zip, then they'll end up with two copies.  But that's better than a really complicated and brittle round of exception handling.

Actually it's worse, as then Rubble will find two addons with the same name and abort. I'll have working automatic-update soon (powered by an addon.meta key and a new AXIS version), so it's probably better to just wait and not worry about it right now.

[Abadrausar]

...Damn... I have no idea why that doesn't work...

I wanted to wait until the webload system were usable before reporting more bugs, but if you are going to rewrite all the AXYS system, and considering that the two problems are probably related to the elusive frontiers between real and mounted virtual filesystem:
Report of a bug: ANY zipped hierarchy of addons should actually be interpreted by the AXYS system in Rubble exactly the same, independently of if it is zipped or unzipped, when the path of access real (unzipped) or virtual (zipped) are the same by means of the tags->name of the root directory of the addonpack in the addon.meta file.
This is not actually true.
Please ask me more detailed info if you need it to clearly identify the bug described.

[milo christiansen]

OK, I figured out what was wrong with the webload system (there was a goof in the "Create" method in the "Multiplexer", which is only used by the addons directory), but I probably won't actully upload a fix, as Rubble 8 (with the all new loader and AXIS 2) is almost ready. Currently it is "finished", but I want to do more testing.

As a special treat you can have a log See if you can spot the newest features... (the random startup line is unintentionally awesome!)

Code: [Select]

Rubble v8.0.0 Internal Test Build
Long hours of debugging later...
==============================================
Attempting to Read Config File: ./rubble.ini
  Read OK, loading options from file.
==============================================
Loading Addons...
  Loading Addons From Local Directories...
  Loading Addons From Local and Remote Addon Packs...
  Loading Pack: FL-Mawrings
    Trying to find update URL...
      No update URL found.
  Loading Pack: The Earth Strikes Back!
    Trying to find update URL...
      No update URL found.
  Checking Loaded Data...
==============================================
Creating New State...
  Loading Special Global Files...
    Reading DF Init Files...
  Creating Script Runners...
==============================================
Starting the Default Web Browser...
  Attempting to Open: "http://127.0.0.1:2120/menu"
Starting Server...
UI Transition: "/menu"
==============================================
Creating New State...
  Loading Special Global Files...
    Reading DF Init Files...
  Creating Script Runners...
UI Transition: "/tsetaddons"
Toggling addon: "Tilesets/MLC/TWBT - 24px" to true
UI Transition: "/tsetrun"
==============================================
Entering Tileset Mode for Region: raw
==============================================
Creating New State...
  Loading Special Global Files...
    Reading DF Init Files...
  Creating Script Runners...
==============================================
Activating...
  Clearing Leftover Addon State Information...
  Loading Config Variables...
    No variables specified.
  Generating Active Addon List...
  Activating Addons from Generated List...
  Pruning Library Addons...
  Activating Required Addons from Meta Data...
  Running Loader Scripts...
    aaa_init_scripting.load.lua
  Active Addons:
    Tilesets/MLC/TWBT - 24px
    Libs/Base
  Checking for Incompatible Addons from Meta Data...
  Setting Unset Config Variables to Their Defaults...
    Attempting to load rubble/userconfig.ini...
      population_cap
      strict_population_cap
      baby_cap_percent
      baby_cap_absolute
      visitor_cap
    Setting Remaining Variables from Meta-Data...
    (Any variables not listed above were explicitly set elsewhere.)
==============================================
Applying Tileset...
  Loading Rules Files...
    aaa_libs_base.tset.rules
  Loading Tileset Files...
    aaa_libs_base.tset.rbl
    tilesets_mlc_twbt.tset.rbl
  Applying Tileset Information...
    b_detail_plan_default.txt
    body_default.txt
    body_rcp.txt
    building_custom.txt
    c_variation_default.txt
    creature_amphibians.txt
    creature_annelids.txt
    creature_birds.txt
    creature_birds_new.txt
    creature_bug_slug_new.txt
    creature_desert_new.txt
    creature_domestic.txt
    creature_equipment.txt
    creature_fanciful.txt
    creature_insects.txt
    creature_large_mountain.txt
    creature_large_ocean.txt
    creature_large_riverlake.txt
    creature_large_temperate.txt
    creature_large_tropical.txt
    creature_large_tundra.txt
    creature_mountain_new.txt
    creature_next_underground.txt
    creature_ocean_new.txt
    creature_other.txt
    creature_reptiles.txt
    creature_riverlakepool_new.txt
    creature_small_mammal_new.txt
    creature_small_mammals.txt
    creature_small_ocean.txt
    creature_small_riverlake.txt
    creature_standard.txt
    creature_subterranean.txt
    creature_temperate_new.txt
    creature_tropical_new.txt
    creature_tundra_taiga_new.txt
    descriptor_color_standard.txt
    descriptor_pattern_iris_eye.txt
    descriptor_pattern_pupil_eye.txt
    descriptor_pattern_special.txt
    descriptor_shape_standard.txt
    entity_default.txt
    inorganic_metal.txt
    inorganic_other.txt
    inorganic_stone_gem.txt
    inorganic_stone_layer.txt
    inorganic_stone_mineral.txt
    inorganic_stone_soil.txt
    interaction_standard.txt
    item_ammo.txt
    item_armor.txt
    item_food.txt
    item_gloves.txt
    item_helm.txt
    item_pants.txt
    item_shield.txt
    item_shoes.txt
    item_siegeammo.txt
    item_tool.txt
    item_toy.txt
    item_trapcomp.txt
    item_weapon.txt
    language_DWARF.txt
    language_ELF.txt
    language_GOBLIN.txt
    language_HUMAN.txt
    language_SYM.txt
    language_words.txt
    material_template_default.txt
    plant_crops.txt
    plant_garden.txt
    plant_grasses.txt
    plant_new_trees.txt
    plant_standard.txt
    reaction_other.txt
    reaction_smelter.txt
    tissue_template_default.txt
  Applying init Patches...
  Running Tileset Scripts...
    tilesets_mlc_twbt.tset.lua
==============================================
Writing Files...
  Writing Raw Files...
    b_detail_plan_default.txt
    body_default.txt
    body_rcp.txt
    building_custom.txt
    c_variation_default.txt
    creature_amphibians.txt
    creature_annelids.txt
    creature_birds.txt
    creature_birds_new.txt
    creature_bug_slug_new.txt
    creature_desert_new.txt
    creature_domestic.txt
    creature_equipment.txt
    creature_fanciful.txt
    creature_insects.txt
    creature_large_mountain.txt
    creature_large_ocean.txt
    creature_large_riverlake.txt
    creature_large_temperate.txt
    creature_large_tropical.txt
    creature_large_tundra.txt
    creature_mountain_new.txt
    creature_next_underground.txt
    creature_ocean_new.txt
    creature_other.txt
    creature_reptiles.txt
    creature_riverlakepool_new.txt
    creature_small_mammal_new.txt
    creature_small_mammals.txt
    creature_small_ocean.txt
    creature_small_riverlake.txt
    creature_standard.txt
    creature_subterranean.txt
    creature_temperate_new.txt
    creature_tropical_new.txt
    creature_tundra_taiga_new.txt
    descriptor_color_standard.txt
    descriptor_pattern_iris_eye.txt
    descriptor_pattern_pupil_eye.txt
    descriptor_pattern_special.txt
    descriptor_shape_standard.txt
    entity_default.txt
    inorganic_metal.txt
    inorganic_other.txt
    inorganic_stone_gem.txt
    inorganic_stone_layer.txt
    inorganic_stone_mineral.txt
    inorganic_stone_soil.txt
    interaction_standard.txt
    item_ammo.txt
    item_armor.txt
    item_food.txt
    item_gloves.txt
    item_helm.txt
    item_pants.txt
    item_shield.txt
    item_shoes.txt
    item_siegeammo.txt
    item_tool.txt
    item_toy.txt
    item_trapcomp.txt
    item_weapon.txt
    language_DWARF.txt
    language_ELF.txt
    language_GOBLIN.txt
    language_HUMAN.txt
    language_SYM.txt
    language_words.txt
    material_template_default.txt
    plant_crops.txt
    plant_garden.txt
    plant_grasses.txt
    plant_new_trees.txt
    plant_standard.txt
    reaction_other.txt
    reaction_smelter.txt
    tissue_template_default.txt
  Writing Tileset Files...
    MLC 24x24 - Graphics.tset.png
    MLC 24x24 - Overrides.tset.png
    curses_1280x600.tset.png
  Writing Init Files...
    init.txt
    d_init.txt
    overrides.txt
      Assembling TWBT Overrides File...
        tilesets_mlc_twbt.tset.twbt
      Writing File.
Run time:  343.5461ms
Done.
UI Transition: "/log"
UI Transition: "/kill"

Anyone who wants to get a jump on the new version can add a file named "pack.meta" to the root of any addon packs that should allow auto-update. This file should be JSON encoded just like addon.meta. There is only one key that I feel comfortable revealing at this point: "Update" should be a URL to your packs download, just like you would use for a .webload file.

Speaking of .webload: Both it and the relatively recent "Downloads" addon.meta key will be going away, as they conflict with the new loader. "Downloads" will be replaced with a key in pack.meta, and .webload is replaced by the "Update" key.

Now is the time to suggest any major changes, as I want to do any breaking that is needed in the 7->8 transition rather than later...

Report of a bug: ANY zipped hierarchy of addons should actually be interpreted by the AXYS system in Rubble exactly the same, independently of if it is zipped or unzipped, when the path of access real (unzipped) or virtual (zipped) are the same by means of the tags->name of the root directory of the addonpack in the addon.meta file.
This is not actually true.
Please ask me more detailed info if you need it to clearly identify the bug described.

I am not quite sure what you mean here...

AXIS should treat a zip file more-or-less exactly like a read-only directory (with child directories and such), any other behavior is a bug that I want to know about. (BTW: I have never used AXIS to read from a zip outside of the addon loader, so if the addon loader doesn't use a feature it is untested).

[Abadrausar]

Now is the time to suggest any major changes, as I want to do any breaking that is needed in the 7->8 transition rather than later...

Things that many of my rubble converted mood needs:

1. Rubble support for the data/speech directory.
2. Rubble support for worldgen and embark files.
3. Enhancing the SO insert with sensible defaults, proposed algoritm:

Spoiler (click to show/hide)

I have modified dev_so_insert.pre.lua in order to give Items and Reactions ADDON_HOOK_PLAYABLE by default, For reactions it is easy because only requires to change the line:

Code: [Select]

{rubble.filters.building, "!SHARED_", {"BUILDING_WORKSHOP", "BUILDING_FURNACE"}, ";NULL", ""},To:

Code: [Select]

{rubble.filters.building, "!SHARED_", {"BUILDING_WORKSHOP", "BUILDING_FURNACE"}, ";ADDON_HOOK_PLAYABLE", ""},For Items is more complicated because I dont see where to add the {!ITEM_CLASS;ADDON_HOOK_PLAYABLE} required after the final bracket of the Item so I do manually.
Could !SHARED_ITEM be enhanced to take an optional default entity as reaction does?

Better than that would be this proposed algoritm in pseudocode:

Code: [Select]

Foreach entity in the converted mod
  build a [key,value] general dictionary (lua table or list) containing the items and reactions enabled in the entity with the format ITEM:IID1 , ... ,ITEM:IIDn ,REACTION:RID1, ... ,REACTION:RIDm
  Where the keys would be formed by the format described and the values the entity that support the Item or Reaction (only exists a general dictionary that contains the items and reactions supported by each separate entity)
Foreach object in Items and Reactions inside the converted mod
   ADD the corresponding {!ITEM_CLASS;ADDON_HOOK_entity...} or {REACTION_ADD_CLASS;ADDON_HOOK_entity...} where it is sound using the generated dictionary.
Maybe I am being too LISPY there...
This could help to make almost completely automatic any Vanilla change of version, and also would make vanal most rubble porting process.

4. I would like to be able to convert the directory addons/Libs in a zip able of autoupdating
5. The line numbers where Opening brace count does not match closing brace count for that line are important to be able of correcting the bad output, disabled raw tags, or simply literal braces in text.
6. Enhancing the rubble lexer to take into account the nested key tokens:

Spoiler (click to show/hide)

Most of DF tokens are values (similar to files) but a few are keys (similar to directorys ) as BODY_DETAIL_PLAN, BODY, BUILDING_WORKSHOP, CREATURE_VARIATION, CREATURE, COLOR, COLOR_PATTERN, SHAPE, ENTITY, INORGANIC, ITEM_AMMO, ITEM_ARMOR, ITEM_FOOD, ITEM_GLOVES, ITEM_HELM, ITEM_PANTS, ITEM_SHIELD, ITEM_SHOES, ITEM_SIEGEAMMO, ITEM_TOOL, ITEM_TOY, ITEM_TRAPCOMP, ITEM_WEAPON, TRANSLATION, SYMBOL, WORD, MATERIAL_TEMPLATE, PLANT, REACTION or TISSUE_TEMPLATE; some key DF tokens have been forfotten like WORLD_GEN and PROFILE only because they occur in unusual places like the data/init directory; others are nested keys inside another root object and the value tokens after them are attached to them until the next apparition of another key token nested or not, you can clearly identify them looking at the indentation of the original vanilla files; an almost exhaustive list of those nested key tokens in vanilla: BP, CV_CONVERT_TAG, BODY_DETAIL_PLAN, ATTACK, CASTE, SELECT_CASTE, WEAPON, POSITION, TISSUE_STYLE, NOUN, ADJ, VERB, PREFIX, SYNDROME, USE_MATERIAL_TEMPLATE, GROWTH, REAGENT and PRODUCT; this is a list of nested tokens that should be flagged as they effectively create nested subobjects inside their root objects; rubble has been partially addressing the symptoms of that (castes and positions independent subsystems and ignoring the rest) instead of addresing the global problem: there exist key (as in directory) nested objects in DF raws and it would be more  simple to give then an uniform and global treatement instead of multiple more hard to maintain or incomplete subsystems...

Milo: Please download: http://dffd.bay12games.com/file.php?id=11963 generate with rubble, and have a look, then you can have an idea of what I am trying to do, I would like some feedback from you about how the system could be done better.

I am not quite sure what you mean here...

The VANILLA library addon is central and their DF objects unmodifiable by the other mods, except by means of the delta semantic operators, inside the vanilla directory  3 visual addons with base mods and 1 hidden library are defined, if you try to convert this vanilla directory into a functionally equivalent zip, you will understand what I mean...

If you have a look, could you please say me what I have done wrong with local sodelta lua definition? it gives me a strange error message, this function concentrates the behavior of {SHARED_OBJECT_KILL_TAG;<ID>;<TAG>}, {SHARED_OBJECT_REPLACE_TAG;<ID>;<TAG>;<REPLACEMENT>} and {SHARED_OBJECT_ADD;<ID>;<PRERAWS>} and logs a detailed info about any unapplied delta to the modder, it is a little ambitious for my level, but probably not for yours, it uses contextual info (Lua globals) to be able to apply a serie of changes to the same object (for example an entity)
While not a perfect system I am working around the patch conflicts between mods like this:
1. I use the addon.meta "LoadPriority": value, to indicate the required order of application of mods.

Spoiler (click to show/hide)

In resume: when a mod applies changes to some part of vanilla I calculate those changes in relation to what is already done by the modest mod (making button the subsidiary maintainer of vanilla raws after Toady one, no problem as soon as both workings are perfectly separated and distinguishable), generalizing this process to other mods you have a formal model of building mods upon the hard work of your precursors instead of fighting them.

2. Then I use delta operators (aka semantic diff) as defined in http://martinfowler.com/bliki/SemanticDiff.html

Code: [Select]

So, while textual diff works with a context of files and lines inside them to define where to apply the delta.
my DF aware delta operators define this context in terms of DF rubble objects and tokens inside them to define where to apply the deltas.
This makes the system more reliable, as you will modify the targetted object independently of in which file it is being defined3. The temporal definition (local non committed version) of those operators is:

Code: [Select]

-------------------------------------------------------------------------------------------------------------------------------------------
-- Semantic Delta operators
-- {@CONTEXT;objectid;wheretoappend}
-- Fixes the objectid (for example ENTITY:MOUNTAIN) where the delta will be applied and also if the appended preraws go before or after (this is the default) the attachment token.
-- {@DELTA;token;value;preraws}
-- Modifies the end value of a token inside an object and  eventually attach before or after it the preraws, if the token id is given but not the value the token is killed.
-- if not token is given the preraws are appended at the end of the object. When no object is found a detailed DELTA warning is generated.
-------------------------------------------------------------------------------------------------------------------------------------------
___rubble_object_cursor = ""
___rubble_attach_after = true
local socontext = [[
local value, after = rubble.targs({...}, {"", "AFTER"})
___rubble_object_cursor = rubble.expandvars(value, "$", true)
if after == "BEFORE" then
___rubble_attach_after = false
else
___rubble_attach_after = true
end
]]
rubble.template("!CONTEXT", socontext)
rubble.template("@CONTEXT", socontext)
rubble.template("CONTEXT", socontext)
rubble.template("#CONTEXT", socontext)
rubble.template("#C", socontext)
local sodelta = [[
local token, value, preraws = rubble.targs({...}, {"", "", ""})
if token == "" then
local data = rubble.registry["Libs/Base:SHARED_OBJECT_ADD"].table
data[___rubble_object_cursor] = (data[___rubble_object_cursor] or "").."\n\t"..rubble.parse(preraws)
return
end
token = string.split(token, ":")
rubble.libs_base.sharedobject_walk(___rubble_object_cursor, function(tag)
if rubble.libs_base.matchtag(tag, token) then
if value == "" then
local preraws = "-"..tag.ID
for _, v in ipairs(tag.Params) do
preraws = preraws..":"..v
end
tag.Comments = preraws.."-"..tag.Comments
else
local fulltoken = "["..token..":"..value.."]"
if ___rubble_attach_after == true then
tag.Comments = preraws..fulltoken..tag.Comments
else
tag.Comments = fulltoken..preraws..tag.Comments
end
end
tag.CommentsOnly = true
else
rubble.warning("OBJECT: "..___rubble_object_cursor.." NOT found, {@DELTA;"..token..";"..value.."} mal formed or incorrect order of application of mods")
end
end)
]]
rubble.template("!DELTA", sodelta)
rubble.template("@DELTA", sodelta)
rubble.template("DELTA", sodelta)
rubble.template("#DELTA", sodelta)
rubble.template("@D", sodelta)4. As they log with enough detail each DELTA failure, modders are guaranted in absence of these warning that ALL intended changes to the vanilla raws have been effectively applied in the order specified by the mods and their dependencies.
5. Some examples of use of the actual DELTA operator applying some changes in vanilla required by the modest mod maintained by Button:

Spoiler (click to show/hide)

-- less fuel required to do a bar of iron
{@CONTEXT;REACTION:PIG_IRON_MAKING}
{@DELTA;REAGENT:A;300:BAR:NO_SUBTYPE:METAL:IRON}
{@DELTA;PRODUCT:100;2:BAR:NO_SUBTYPE:METAL:PIG_IRON}
-- less fuel required to do a bar of steel
{@CONTEXT;REACTION:STEEL_MAKING}
{@DELTA;REAGENT:A;300:BAR:NO_SUBTYPE:METAL:IRON}
{@DELTA;REAGENT:B;300:BAR:NO_SUBTYPE:METAL:PIG_IRON}
{@DELTA;PRODUCT:100;4:BAR:NO_SUBTYPE:METAL:STEEL}

Some mods depends on Modest mod that is almost complete but not finished, and modest  mod depends on working delta operators but less complete and functional that the one that I have asked your help to debug.

[Dirst]

Anyone who wants to get a jump on the new version can add a file named "pack.meta" to the root of any addon packs that should allow auto-update. This file should be JSON encoded just like addon.meta. There is only one key that I feel comfortable revealing at this point: "Update" should be a URL to your packs download, just like you would use for a .webload file.

Speaking of .webload: Both it and the relatively recent "Downloads" addon.meta key will be going away, as they conflict with the new loader. "Downloads" will be replaced with a key in pack.meta, and .webload is replaced by the "Update" key.

Now is the time to suggest any major changes, as I want to do any breaking that is needed in the 7->8 transition rather than later...

That's almost exactly what I was envisioning, thanks!  Does it do a pre-pass thru all of the zips first, then load everything for real?

As for major changes, I would really like to find some way of supporting third (fourth?) party content like Stonesense assets, Armok Vision models, Soundsense packs, Therapist grids, etc.  As a starting point, these could be considered bricks that the engine more or less blindly unzips at the target location (just confirm to overwrite).  Something BAMMy might eventually make sense, but I wouldn't want you to spend a god-awful amount of time with special-case-hell just to get the initial version out the door.

Another suggestion would be to get Internet at your home

[Abadrausar]

Another suggestion would be to get Internet at your home

I am sure Milo would be happy to have a working Internet If he could....

Dirst: Could you download that and have a look? http://dffd.bay12games.com/file.php?id=11963

It would be nice to have some feedback or opinions.

[milo christiansen]

Does it do a pre-pass thru all of the zips first, then load everything for real?

Nope, it loads the first pack and any others it requests first, then continues down the list recursively (skipping any it has already loaded). Addons in directories are loaded before packs, just like before. In general addons in directories should be reserved for WIP stuff and the addons that ship with Rubble.

As for major changes, I would really like to find some way of supporting third (fourth?) party content like Stonesense assets, Armok Vision models, Soundsense packs, Therapist grids, etc.

The trouble with "global" content like this is that it falls outside of the norm. It is a good idea, but I am not sure how best to handle it... Maybe a way to flag a directory and its children (including child directories) as a "copy file addon" that is then copied to a given AXIS path during the write stage?

1. Rubble support for the data/speech directory.

This is tricky. I don't want Rubble writing anything that cannot be moved by simply packing up the save, but unfortunately some mods need to write things elsewhere.

Currently it is possible to write files to this directory (via the AXIS path "df:data/speech" (or "df/data/speech" in v8)), but this is a minor pain. I suppose a trivial worry about not having self-contained output in one minor case is no good reason to not support it better. Look for a new first-part extension and file tag supporting writes to this directory in v8.

2. Rubble support for worldgen and embark files.

Same deal as data/speech, but with less reluctance on my part. I suppose something made with the raw parser that allows only modifying/adding named objects would be what is wanted?

3. Enhancing the SO insert with sensible defaults...

Good ideas, easy to do. Look for it in v8.

4. I would like to be able to convert the directory addons/Libs in a zip able of autoupdating

No. Too much stuff there depends on internal details of the current Rubble version. Library addons should be updated at the same time as the Rubble binary. When I add auto-update for Rubble itself this will be included as part of that.

5. The line numbers where Opening brace count does not match closing brace count for that line are important to be able of correcting the bad output, disabled raw tags, or simply literal braces in text.

This isn't quite clear. You want line numbers for where the brace counts don't match? I do too, but it is impossible for Rubble to calculate this with any accuracy (how can you tell which brace isn't paired? To put it simply, you can't.)

6. Enhancing the rubble lexer to take into account the nested key tokens:

Big problem here: Rubble has no concept of a DF "object", much less a sub-object. The Rubble template parser totally ignores raw tags, instead dealing with each file as a chunk of unstructured text. The only thing the template parser cares about is templates. Rubble does an OK job of faking a more sophisticated approach via the shared object system, but really even that looks at the object body simply as a block of text.

The main problem is that a Rubble raw file may or may not look like raws, depending on how extensive use of templates is. Every time I try to make something "smarter" it runs straight into the fact that, until after the final parse stage, a Rubble raw file is not really readable as raws. The body of a shared object is often fairly readable, but only because most of the time it has few templates. This is not always the case, leading to "smart" editing templates being unreliable.


@Abadrausar: I'll take a look at your project, it looks interesting...

The main problem I forsee is that it may be too complicated to maintain easily and/or too hard to convert existing mods for those unfamiliar with Rubble. The system itself looks very interesting, but I want to play with it some before I say much.

[Abadrausar]

5. The line numbers where Opening brace count does not match closing brace count for that line are important to be able of correcting the bad output, disabled raw tags, or simply literal braces in text.

This isn't quite clear. You want line numbers for where the brace counts don't match? I do too, but it is impossible for Rubble to calculate this with any accuracy (how can you tell which brace isn't paired? To put it simply, you can't.)

The problem there is Masterwork, they delete one of the brackets of a token but not the other and that causes undefined behavior, they have some exceptional and unique content that I would like to reuse, but if I am not able of identifying which lines have the problem and correct it ... Reading the raws it can be seen that almost never the brackets in a line are unbalanced. It can be that if the utility is developed it could detect some false positives but inversely it is clear that the rest are the lines with the problem. I have tried to detect the unbalanced brackets myself reading the code but I lose soon my concentration

6. Enhancing the rubble lexer to take into account the nested key tokens:

Big problem here: Rubble has no concept of a DF "object", much less a sub-object. The Rubble template parser totally ignores raw tags, instead dealing with each file as a chunk of unstructured text. The only thing the template parser cares about is templates. Rubble does an OK job of faking a more sophisticated approach via the shared object system, but really even that looks at the object body simply as a block of text.
The main problem is that a Rubble raw file may or may not look like raws, depending on how extensive use of templates is. Every time I try to make something "smarter" it runs straight into the fact that, until after the final parse stage, a Rubble raw file is not really readable as raws. The body of a shared object is often fairly readable, but only because most of the time it has few templates. This is not always the case, leading to "smart" editing templates being unreliable.

Well at least we have compiled a list with the offensive tokens, maybe after some reflex ion someone could have a sound idea about the possible implementation:
Most of DF tokens are values (similar to files) but a few are keys (similar to directorys ) as BODY_DETAIL_PLAN, BODY, BUILDING_WORKSHOP, CREATURE_VARIATION, CREATURE, COLOR, COLOR_PATTERN, SHAPE, ENTITY, INORGANIC, ITEM_AMMO, ITEM_ARMOR, ITEM_FOOD, ITEM_GLOVES, ITEM_HELM, ITEM_PANTS, ITEM_SHIELD, ITEM_SHOES, ITEM_SIEGEAMMO, ITEM_TOOL, ITEM_TOY, ITEM_TRAPCOMP, ITEM_WEAPON, TRANSLATION, SYMBOL, WORD, MATERIAL_TEMPLATE, PLANT, REACTION or TISSUE_TEMPLATE; some key DF tokens have been forfotten like WORLD_GEN and PROFILE only because they occur in unusual places like the data/init directory; others are nested keys inside another root object and the value tokens after them are attached to them until the next apparition of another key token nested or not, you can clearly identify them looking at the indentation of the original vanilla files; an almost exhaustive list of those nested key tokens in vanilla: BP, CV_CONVERT_TAG, BODY_DETAIL_PLAN, ATTACK, CASTE, SELECT_CASTE, WEAPON, POSITION, TISSUE_STYLE, NOUN, ADJ, VERB, PREFIX, SYNDROME, USE_MATERIAL_TEMPLATE, GROWTH, REAGENT and PRODUCT; this is a list of nested tokens that should be flagged as they effectively create nested subobjects inside their root objects; rubble has been partially addressing the symptoms of that (castes and positions independent subsystems and ignoring the rest) instead of addresing the global problem: there exist key (as in directory) nested objects in DF raws and it would be more  simple to give then an uniform and global treatement instead of multiple more hard to maintain or incomplete subsystems...

@Abadrausar: I'll take a look at your project, it looks interesting...
The main problem I foresee is that it may be too complicated to maintain easily and/or too hard to convert existing mods for those unfamiliar with Rubble. The system itself looks very interesting, but I want to play with it some before I say much.

It is more easy that it seems when you follow the good set of conventions or rules, what is hard is finding those...Dark Ages II were working like a charm before I sabotaged myself the Delta operators trying to made them almost perfect and Modest mod is also 90% done. Complete list of Mods and addons of the unreleased  RC2:

Spoiler (click to show/hide)

_________________________________________   _______________________________Composable_Base_mods________________________________________________
Toady_One___Vanilla   Toady One Vanilla base raw files, rubble porting by Abadrausar, able of fine grained imports.
Button___Modest   Collection of vital Vanilla bugfixes and tweaks that everyone should be able to use comfortably and without reservation.
GM-X___Dark_Age_II   Dark Age has fallen upon the world of Dwarf Fortress.
Abadrausar___Haiku_Extreme   DF in 41 active Vanilla objects with only 6 modified and all the features.
Abadrausar___Haiku_Vanilla   DF in 65 active Vanilla objects without any modification and all the features.
Abadrausar___Singularity   The entire multiverse is collapsing into a single singular world.
IndigoFenix___Rise_of_the_Mushroom_Kingdom   Rise of the Mushroom Kingdom
Milo___First_Landing   SCI Themed 4x, explore and colonize foreign planets.
Putnam___Sparking   Complete dragonball lore.
Roses___The_Third_Age   We now live in the Third Age, and new threats abound.
Valikdu___My_Little_Fortress   Attempt at expanding and enhancing the original MLP mod by Nidokoenig.
__________________________________________   _______________________________Composable_mods_that_require_some_Base_mod_______________________
Abadrausar___Humble   FPS aware set of minimods
Darkflagrance___Fortress_Defense   More concentrated FUN
Dirst___The_earth_strikes_back   Any miner knows that the surest way to prosperity is to strike the earth. Wise miners know that sometimes the earth strikes back.
IndigoFenix___Mostly_mythical_monsters   Mostly mythical monsters
IndigoFenix___Undertale   This mod adds the monsters from the game Undertale as a playable race
Lovechild___All_races_playable   You can play as Vanilla dwarf, elf, human, goblin, kobold, or animalpeople.
Putnam___Fantastic_MiniMods   Fantastic_MiniMods includes Materials Plus, Microreduce and Secrets n curses
Rhenaya___Drow   Drow civilization and lore.
Wannabehero___Deep_Dwarven_Domestication   Diverse and thematic domestic animals. It also enables generic honey, the use of nail as a kind of horn, chitin as shell and feathers as pearls.
___________________________________________   ______________________(Not_Composable_mods)incompatibles_with_all others____________________
Milo___First_Landing_FINAL   Makes First Landing final, incompatible with other bases because it removes candy and anticandy.
____________________________________________   ______________________________Modders_Toolbox________________________________________________
Milo___Rubblize_Raws   Adds SHARED_OBJECT tags to non-Rubble raws, semiautomatic only.
Abadrausar___FPS   FPS aware set of minimods
Abadrausar___No_more_giants   Removes all giants from active game, they will appears only in engravings and legends.
Abadrausar___QuickFort   QuickFort aware set of minimods
Dirst___Accessibility   A set of simple commands for the DFHack console to aid in navigating the map. Comments and suggestions are welcome.
Milo___Cheat_Reactions   A bunch of cheat reactions to allow for fast setup of testing forts.
Milo___Dummy_Reactions   Empty reactions registered to ADDON_HOOK_PLAYABLE.
Milo___Dummy_Workshops   How many dummy workshops should be generated?
Milo___Generic_Materials   Support for the Generic Animal Materials.
Milo___Generics   QuickFort aware set of minimods
Milo___Graphics_Defense   Activate to keep Rubble from nuking your creature graphics.
Milo___Pop_Cap   Allows you to set a population cap that will override the default for this world.
Putnam___Materials_Plus   Adds a bunch of materials to the game.
Putnam___Microreduce   Reduce some micromanaging needs.
Putnam___Secrets_n_Curses   A mod that adds a few random secrets and curses that didn’t fit in any of my existing mods.
Uncle_Span___Leathery_Vegetables   Huskshroom and Leather-leaf, plants that gives you skin to tan or leather ready leafs
_____________________________________________   ______________________________Tilesets_______________________________________________
Tileset___Bisasam   A simple addon that installs antialiased 20x20 pure ASCII-like Bisasam�s tileset.
Tileset___Uncle_Span   Uncle Span tileset.
Tileset___Yobbo   A simple addon that installs Yobbo modification of antialiased 20x20 pure ASCII-like Bisasam�s tileset.
______________________________________________   ______________________________Graphics_Sets______________________________________________
Graphics___CLA   Graphic set that tries to keep the simplicity and ease-of-recognition of ASCII-like tilesets and add a little beauty and variety of graphic sets.
_______________________________________   _______________________________Tutorials___________________________________________________
________________________________________   _______________________________Tests______________________________________________________
Test/ISS   Intensive subterranean sprouting.
Test/ISS2   Intensive subterranean sprouting.
Test/IntensiveSprouting   some testing3.
Test/test01   some testing.
Test/test02   some testing2.
Test/test03   Intensive subterranean sprouting.
Test/test04   some testing 4.
Test/test05   Intensive subterranean sprouting.
Test/test06   test6
Test/test07   test6
Test/test08   test8
Test/test09   test9
Test/test10   test
Test/test11   test
Test/test12   test
Test/test13   test
Test/test14   test
Test/test15   test
Select All None
Hide/Show DFHack - Dev -
Edit configuration variables Skip
Back to Menu

I am very interested in exploring the Haiku_Vanilla concept:
Minor mods that may have a positive impact over FPS or give bigger relevance to the auto generated content of the simulation: Ruins, Tombs, Lairs, Titans, megabeast, semimegabeast, Bogeyman, vampires,mummies, clowns, ghost, zombies and necros, clowns and Mayor Demons, Gods, magic weather, night creatures… Dwarf Fortress File Depot definition of minor mod is relevant here.
How?
Haiku Vanilla

This is DF with the minimal number of active objects to have all the basics of a functional vanilla without modifying any of them only pruning all the non redundant object into a separate minimal directory nested inside the base of their type.

What we have until now is:

10 Creatures: 1 wagon,3 vermin,1 colonies (bees), 3 domestics (dog, cat and donkey) 2 civilized (dwarves and goblins) and nothing more. For the variety you have the titans, FB semiFB, night creatures and rest of random.
34 Inorganics: 5 to enable adventure coins, 6 for the bronze line, 5 for the steel line, 3 for the deep specials, and 15 for the rest: stones and ores.
21 Plants: 2 grass one subterranean (cave moss) another above ground (meadow-grass), 8 crops: 6 subterranean (Cave wheat, Dimple cup, Pig tail, Plump helmet, Quarry bush and Sweet pod) and 2 above ground (Sliver barb and Whip vine), 11 trees 7 subterranean (Black-cap, Blood thorn, Fungiwood, Goblin-cap, Nether-cap, Spore tree, Tower-cap and Tunnel tube) and 4 above ground (glumprong, feather tree, highwood and willow)
That makes a great total of 65 active objects to play with, nothing more (this is not strictly true) nothing less; AND HEY!, before you begin to protest, remember that this is a minimal mod done in the interest of SCIENCE, it could be FUN to play or not (it is up to you to see and decide), but this is not certainly the unique objective of this mod not even the primary.
The number of active objects have been already dramatically reduced (good news for your FPS) but I think that it could be reduced even further, some more SCIENCE required… When you experiment trying to remove even more objects you will find some of the hard limits of the game.
Vanilla_haiku is also a testbed of the possibilities of the library VanillaLibs to serve the requested content to other mods. 

Some of those hard limits already more or less identified:

It seems that you must have at least two entities defined to elude world generation rejects.
A wagon creature must be define in order to be able to start the game in fort mode.
You need at least the vanilla base language files and then the specifics language files of the creature of your civilization if those creatures are able to speak.
BODY, BODY_DETAIL_PLAN, BOOK, BUILDING, DESCRIPTOR_COLOR, DESCRIPTOR_PATTERN, DESCRIPTOR_SHAPE, INTERACTION, ITEM, MATERIAL_TEMPLATE, REACTION, SECRET and TISSUE_TEMPLATE have not been investigated for pruning.