Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[jecowa]

Graphics extension module organization

Subfolders might be better than putting suffixes on everything. It sounds more simple to copy everything from a certain folder than to copy only the files with certain suffixes. And subfolders would keep the modules better organized too. Even if we use suffixes, I'd like to have the ability to put each module in its own folder if possible.

Graphics extensions modifying objects files

While thinking about how to add graphics support for Arctic Additions, I realized that graphics modules will need to be able to edit the /raw/objects/ of the mod in order to change which tiles get used and the colors. Most if not all of the more graphical tilesets move a bunch of stuff around in the main tilesheet. Examples:

• Most of them repurpose all the accented characters for graphics use.
• Mayday has some kind tiered-stone system and assigns one of three tiles to a stone based on some criteria.
• Ironhand has "hybrid" tiles that contain two sprites in one tile.

A plant in a mod like Arctic Additions would need to be able to specify different tiles depending one what graphics pack is being used, since that tile could very likely be in a different location in every tileset. So it would be helpful for graphics extensions to edit the objects files.

[Meph]

You can circumvent that by using TWBT overrides for the plant. Regardless of how the tilesets look you use, the mod graphics for the plant would always look the same.

[jecowa]

Graphics modules won't support overrides until multiple overrides.txt sheets can be used alongside each other. And it would be nice for graphics to look good for players not using TwbT.

Also, Rydel was saying that overrides might have some problems when used with mods, but I'm not really sure how much of a problem it is.

@Rydel:  items can in fact be referenced by name, so we can avoid the problem of numeric IDs changing.  Perhaps we (or Mifki) should detect use of numbers and issue a deprecation warning?

That link shows names for Ids and Types, but last I heard, subtypes are still numerical.  This would be a problem for mod support since subtypes for existing entries can change when you install a mod.

I don't know what a subtype is. Can I make a TwbT override without using a subtype? (Do "main types" or "super types" exist or something?) If the subtypes can never be known, are overrides completely useless?

[Meph]

Graphics modules won't support overrides until multiple overrides.txt sheets can be used alongside each other.

Why? Can't you merge the override file just like you merge the raws?

Since you can call override tilesets like this:

Code: [Select]

[TILESET:tileset:tileset.png:_any_name_you_want], modders could just use unique names for each set, like "_MEPHS_STUFF_VERSION_1_2".

[jecowa]

If multiple mods try to add overrides, all but one overrides file would get overwritten.

A work-around would be to combine and store the overrides files for every mod for a graphics pack in every graphics extension module for that graphics pack. I'd be fine with it working like that until it's possible to use multiple overrides.txt files.

[Dirst]

Suggested starting point:

• (recap:) A mod in PyLNP format is simply a DF install with the vanilla files removed.  We then merge the raw/ and data/speech/ files.
• To support graphics, mods just include graphics raws and images.  As long as they don't have the same name as a file in the graphics pack, this already works!  (I just checked the code - apparently I thought of this two years ago )
• Support for data/art and data/init/<overrides files> will be added once Mifki supports multiple overrides files.
• (for discussion) How should non-default graphics extensions be stored?  How should PyLNP choose which to install?  Does this work with multiple mods?

FYI, I've never been able to get the mod merging tool to import graphics under Mods/___/raw/graphics/

Code: [Select]

Running PyLNP 0.12 (OS: win, Compiled: True)
WARNING: JSONConfiguration: File PyLNP.user does not exist
ERROR: In "TESB": : file "graphics\graphics_TESB_creatures.txt": : Writing to .\LNP\Baselines\temp\raw\graphics\graphics_TESB_creatures.txt failedAdding that capability would simplify installation.

As for TWBT overrides, the easiest solution (for everyone except mifki) would be to use named subtypes.  With numeric subtypes, it's possible that more than one mod could add subtypes to the same type.  One solution would be to keep a running map of each mod's subtypes to a global list of subtypes (they should each be written as if they were modifying vanilla entries or adding directly to the end of vanilla).  Though conceptually simple, it just feels error-prone and thus would need the option for verbose logging.

Vanilla ARMOR:1 -> ARMOR:1
Vanilla ARMOR:2 -> ARMOR:2
...
Vanilla ARMOR:11 -> ARMOR:11
SuperMod ARMOR:12 -> ARMOR:12
SuperMod ARMOR:13 -> ARMOR:13
UltraMod ARMOR:12 -> ARMOR:14
UltraMod ARMOR:13 -> ARMOR:15
UltraMod ARMOR:14 -> ARMOR:16
XenoMod ARMOR:7 -> ARMOR:7 // Redefining a vanilla object.

Slip the mapped values into the tempfiles, then concatenate them into a single overrides.txt file.

[Meph]

If multiple mods try to add overrides, all but one overrides file would get overwritten.

How do you want to install mods in that case? I was under the impression that the PyLNP merges the raw files. If it would just replace files, you can barely install any two mods at the same time. Anything that changes vanilla DF files would be impossible.

[jecowa]

PyLNP merges things by copying both of their files into the same folder. If two mods try to add a /data/init/overrides.txt, only of those files will survive the merger.

As for TWBT overrides, the easiest solution (for everyone except mifki) would be to use named subtypes.  With numeric subtypes, it's possible that more than one mod could add subtypes to the same type.  One solution would be to keep a running map of each mod's subtypes to a global list of subtypes (they should each be written as if they were modifying vanilla entries or adding directly to the end of vanilla).  Though conceptually simple, it just feels error-prone and thus would need the option for verbose logging.

Vanilla ARMOR:1 -> ARMOR:1
Vanilla ARMOR:2 -> ARMOR:2
...
Vanilla ARMOR:11 -> ARMOR:11
SuperMod ARMOR:12 -> ARMOR:12
SuperMod ARMOR:13 -> ARMOR:13
UltraMod ARMOR:12 -> ARMOR:14
UltraMod ARMOR:13 -> ARMOR:15
UltraMod ARMOR:14 -> ARMOR:16
XenoMod ARMOR:7 -> ARMOR:7 // Redefining a vanilla object.

Slip the mapped values into the tempfiles, then concatenate them into a single overrides.txt file.

It reminds me of the strife specibus from Home Struck (browser game that teaches data structures). It seems like the map would need to know the sub types for every combination of mods. (I think mods can't pick subtypes for their items, but the game assigns them dynamically based on the alphabetical order of something or other.) And that subtype map would probably need to be updated every time any mod adds a new item or maybe renames one or something. I think this task sounds impossible to solve from PyLNP's end. TwbT might be the more logical place to take care of figuring out subtypes, since TwbT has access to DFHack and can probably get the subtype numbers for items from it.

[Dirst]

Correct, mods cannot pick the subtype numbers, but they are assigned in a deterministic order based on the known order in which raws are parsed.  The thing to prevent is someone jumping in front of the vanilla stuff.

[Rydel]

You can circumvent that by using TWBT overrides for the plant. Regardless of how the tilesets look you use, the mod graphics for the plant would always look the same.

Plant overriding in TWBT requires a lot of raw edits.  TWBT on its own can't tell the difference between shrubs, fruit, leaves, dead plants, etc, so all of those need to be overrided in the raws.

I don't know what a subtype is. Can I make a TwbT override without using a subtype? (Do "main types" or "super types" exist or something?) If the subtypes can never be known, are overrides completely useless?

Subtypes are how the game identified items in the same category.
Say we were looking at tools.  All of them have the ID TOOL and the Type TOOL, an the subtype tells which tool it is.
These are automatically numbered in the order listed in the raws, so 0 is Cauldron, 1 is Ladle, 2 is Bowl, etc.

The subtypes are known, but the person writing the override file has to know what order items are in the raws in advance.  Stuff getting added to the end won't mess stuff up, but files getting added at the beginning or in the middle will cause problems.

Let's say a mod added a file called item_mod_tools.txt with a new tool, called NEWTOOL.
Now 0 is NEWTOOL, 1 is Cauldron, 2 is Ladle, 3 is Bowl...
Since your overrides were written without the mod, all the NEWTOOLs will look like cauldrons, the cauldrons will look like ladles, etc.

You can't work without subtypes because (aside from TWBT just throwing an error and probably crash) it would make every look look identical.

As for TWBT overrides, the easiest solution (for everyone except mifki) would be to use named subtypes.

I agree, but I think I've asked about that feature before and it wasn't possible.  I think DFHack doesn't know the names the raws used, it just has the number pulled from the program itself.  For things like Id and Type, those won't change, so they are known at compile time and can be given specific names.

Correct, mods cannot pick the subtype numbers, but they are assigned in a deterministic order based on the known order in which raws are parsed.  The thing to prevent is someone jumping in front of the vanilla stuff.

For preventing files from jumping in front of vanilla stuff, the easiest way I can think of it to prepend all the mod files with "zz_", forcing them to be last alphabetically.

[PeridexisErrant]

Alright, time for another (shorter...) essay.

TwbT Overrides
@Meph @Rydel:  OK, apparently I'll also be asking Mifki to support subtypes by name.  (it would also be good to log a deprecation warning for numerical indicies, both to encourage change and explain to users why things are broken...).  DFHack does have access to the names - look for example at the view-item-info and item-descriptions scripts: the former looks up text in the latter by item type or subtype name.

We can't reliably merge overrides files, as they would not have the level of context that raws generally provide (ie unchanged vanilla lines).  TBH I'm surprised sometimes that the merge works for raws at all...  improving this has been a PyLNP goal for years, but it's a really hard problem.  Unique names so we can just keep adding files is therefore my favorite solution!

Preprocessing files to keep track of item subtype ID numbers... just no.  It's may be technically possible, but it's not happening.  I'd improve raws merging instead if I could do this.


Where do Graphics extensions live
As @Dirst says, it's just be whatever goes in "LNP/Mods/<some_mod>/raw/graphics" - not a new kind of thing (which would be a combinatorial explosion at best).  This is obviously a default-only thing; extensions for other graphics packs will live in subdirectories ("LNP/Mods/<some_mod>/graphics-extensions/<name-of-gfx>/", for example).

@Jecowa - creating a graphics extension for a mod makes you a mod author (or contributor, in which case you do the usual sned-changes-to-author dance); but you don't need the graphics pack to help you or even know graphics extensions exist - so they'll be backwards-compatible

No suffixes or renaming.  It's just to fragile, and prone to breaking in hard-to-diagnose ways!


How does PyLNP decide which graphics extension to install?
The manifest for a mod will have a new attribute "graphics-extensions", which is a dictionary mapping graphics packs to subdirectory names (see above).  Technically, the extension is just another mod to merge (so object files can be changed); by policy it should just override the relevant graphics bits.

PyLNP will install the default mod (which includes graphics), then if the installed graphics pack matches a subdirectory (per the mapping) will install the relevant extension.  Mod authors may therefore wish to provide nothing as the default extension (ie leave /raw/graphics and twbt files nonexistent), to avoid installation clashes.


Other comments
@Meph - we feel exactly the same way about standards; I'll be sure to let you know.  Honestly I'd love to bring PyLNP and the masterwork launcher closer together or even merge them, but that's a conversation for another time and place (and after some upgrades...).

Maybe mods writing to raw/graphics is banned by code elsewhere?  Anyway, the core code handles this OK and I should be able to make it work without too much trouble.

PyLNP copies image files and DFHack scripts, but merges .txt (eg raws) and .init files.  Otherwise we would indeed have (even more) trouble installing more than one mod at a time.

[jecowa]

but you don't need the graphics pack to help you or even know graphics extensions exist - so they'll be backwards-compatible
[...]
How does PyLNP decide which graphics extension to install?
The manifest for a mod will have a new attribute "graphics-extensions", which is a dictionary mapping graphics packs to subdirectory names (see above).

The dictionary map would be a sort of spreadsheet in the mod's manifest that tells PyLNP which graphics extension goes with which graphics packs, right? Can it do a partial match on the graphics pack, in case a user has renamed two copies of Phoebus to things like "Phoebus v0.43.03" and "Phoebus v0.43.05", for example?

Technically, the extension is just another mod to merge (so object files can be changed)

That's good to hear. That's something that's currently not possible with mod support added to graphics packs.

PyLNP will install the default mod (which includes graphics), then if the installed graphics pack matches a subdirectory (per the mapping) will install the relevant extension.  Mod authors may therefore wish to provide nothing as the default extension (ie leave /raw/graphics and twbt files nonexistent), to avoid installation clashes.

That sounds good.

Maybe mods writing to raw/graphics is banned by code elsewhere?

Sorry, I just tried it again (in PyLNP v0.11) and it worked. I was probably looking in the wrong Dwarf Fortress folder when checking to see if it installed. I think I did that at one point and couldn't figure out why nothing would install.

PyLNP [...] merges .txt (eg raws)

It's not working like that for me. From what I can tell, if a user tries to install more than one mod (like Spellcrafts, Haberdasher, and The Earth Strikes Back, for example) that all want to edit to the same file (such as entity_default.txt), only highest-listed of those mods in the "Merged" list of the "Mods" tab will have a green background color (and will installed normally). The lower-listed mods trying to edit that file will have white backgrounds (and will silently not be installed).

I kind of thought the files it really could mess with (other than its own) were init.txt and d_init.txt.

[PeridexisErrant]

Can it do a partial match on the graphics pack, in case a user has renamed two copies of Phoebus to things like "Phoebus v0.43.03" and "Phoebus v0.43.05", for example?

PyLNP [...] merges .txt (eg raws)

It's not working like that for me. From what I can tell, if a user tries to install more than one mod (like Spellcrafts, Haberdasher, and The Earth Strikes Back, for example) that all want to edit to the same file (such as entity_default.txt), only highest-listed of those mods in the "Merged" list of the "Mods" tab will have a green background color (and will installed normally). The lower-listed mods trying to edit that file will have white backgrounds (and will silently not be installed).

I kind of thought the files it really could mess with (other than its own) were init.txt and d_init.txt.

I certainly plan to support a degree of fuzzy matching.  My current favourite test is to lower-case everything (so it's not sensitive to capitalisation), then for each graphics pack check if the ID-to-load is part of the pack name.  This is easy to write and understand, unlike (eg) regular expressions.

If you're seeing unmerged mods *without* an incompatible (red) line above them, please post your logs in the PyLNP thread and we'll investigate there.

[Dirst]

In my experience, a white line usually means there was some kind of error (e.g., unable to copy a graphics txt file).  The log ought to have something about it.

[jecowa]

Also, I think it would cause blank sprites if one graphics set has hunting/war sprites for an animal but another does not.

It would crash at worldgen. If a graphics file directs to an image that does not exist, the game crashes to desktop.

I think it might not crash if the graphics are for creatures that do not exist in the game.