Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Leonidas]

Since the game-killing corruption comes from bad entries in the item list, shouldn't it be possible to write a script to comb through that list and detect those entries? Even if it weren't possible to delete them, it would help to at least know that the fort had been corrupted.

[PatrikLundell]

I've looked for bad items in the item list and haven't found them. The problem, from the outside, is that the corruption is seen by one or more equipment lists containing garbage entries, but those are generated on the fly (I've tried hacking them to remove a bad entry, and it's back the next time the list is generated). To me, it seems that the structure the data comes from isn't the regular item lists, but something else, unknown what (Toady would know, but I don't think anyone has asked him).

If we knew where the data came from (and it has been mapped by DFHack), we'd be able to do as you propose (I've thought about it as well, but the missing knowledge of where the source is shoots that idea down). It might be possible to write a script that automatically brings up the item lists and then scans them, though (not something I've done).

[lethosor]

Yeah, military equipment (assuming that's what "equipment" is referring to) is likely stored separately from the items themselves.
Were you modifying the equipment list in a viewscreen? Those are indeed usually temporary copies. I'm not familiar with the exact structures at play here, but I could take a look at one of the corrupted saves.

[Roses]

I've looked for bad items in the item list and haven't found them. The problem, from the outside, is that the corruption is seen by one or more equipment lists containing garbage entries, but those are generated on the fly (I've tried hacking them to remove a bad entry, and it's back the next time the list is generated). To me, it seems that the structure the data comes from isn't the regular item lists, but something else, unknown what (Toady would know, but I don't think anyone has asked him).

Yeah, military equipment (assuming that's what "equipment" is referring to) is likely stored separately from the items themselves.
Were you modifying the equipment list in a viewscreen? Those are indeed usually temporary copies. I'm not familiar with the exact structures at play here, but I could take a look at one of the corrupted saves.

I've looked in the past, but I've never been able to find the root location of the items. I'm thinking that somewhere there is a list of types, subtypes, mat_types, and mat_subtypes and that for one reason or another one of the four gets incorrectly filled in (or maybe not filled in) and then when the equipment lists get filled from those lists the corruption occurs.

My thought for why it is stored this way is because that is how the entity resources are stored, and I have had corrupted entity lists when I have accidentally removed a subtype entry and not the corresponding type entry.

[PatrikLundell]

@lethosor: Yes, I modified the viewscreen lists, because that was how far back I could trace it. I wasn't very surprised that it didn't work, though, but since that was the only location known I though I'd at least try it.

[Leonidas]

I've looked in the past, but I've never been able to find the root location of the items. I'm thinking that somewhere there is a list of types, subtypes, mat_types, and mat_subtypes and that for one reason or another one of the four gets incorrectly filled in (or maybe not filled in) and then when the equipment lists get filled from those lists the corruption occurs.

If the problem is with the system that turns general item types into specific items, (e.g. "mace" becomes "+steel mace+"), then couldn't you avoid the corruption by never assigning uniforms and only giving soldiers specific items?

[PatrikLundell]

I've looked in the past, but I've never been able to find the root location of the items. I'm thinking that somewhere there is a list of types, subtypes, mat_types, and mat_subtypes and that for one reason or another one of the four gets incorrectly filled in (or maybe not filled in) and then when the equipment lists get filled from those lists the corruption occurs.

If the problem is with the system that turns general item types into specific items, (e.g. "mace" becomes "+steel mace+"), then couldn't you avoid the corruption by never assigning uniforms and only giving soldiers specific items?

No. The crash happens (presumably) when DF scans the list for equipment to give to troops, regardless of what the equipment assigned is. One possible issue is checking the size of a book to see if it can be worn by a dwarf, as books don't have that kind of size information, so it would probably result in a crash when trying to access a non existent field, nor does any field for clothing layers exist, etc.

[Leonidas]

The crash happens (presumably) when DF scans the list for equipment to give to troops, regardless of what the equipment assigned is. One possible issue is checking the size of a book to see if it can be worn by a dwarf, as books don't have that kind of size information, so it would probably result in a crash when trying to access a non existent field, nor does any field for clothing layers exist, etc.

Why would it scan a list when the user has already chosen the specific item to be equipped?

There have always been bugs surrounding the system that turns the archer uniform into a "leather headware" assignment, which becomes a specific *wombat leather helm*. And it sounds like the introduction of books made those bugs worse. So bypass the whole rotten system. Create the squads with No Uniform, and equip the soldiers one specific item at a time, so that their equipment screen looks like this:

Spoiler (click to show/hide)

It would be tedious, of course. But if the concept worked, then maybe someone could write a utility to streamline the process.

[PatrikLundell]

I suggest you take one of the corrupted saves on DFFD, disbands all squads, and creates new ones with only specific items for each member (in particular containing items from the corrupted categories, although you really should allocate all normal slots), and run it for at least 3 years, and you should also produce new equipment of kinds used (to try to force evaluations and see that they're not performed, basically). If that succeeds you've proven your concept. Personally I think it will fail, but I'll be happy to be proven wrong.

[Leonidas]

I suggest you take one of the corrupted saves on DFFD, disbands all squads, and creates new ones with only specific items for each member (in particular containing items from the corrupted categories, although you really should allocate all normal slots), and run it for at least 3 years, and you should also produce new equipment of kinds used (to try to force evaluations and see that they're not performed, basically). If that succeeds you've proven your concept. Personally I think it will fail, but I'll be happy to be proven wrong.

I'm not sure that this approach would eliminate corruption after it starts. My plan is to try it on a new fort where no soldier has ever been assigned non-specific equipment. I don't know much about the technical side, but wouldn't it demonstrate something if a fort like that ran smoothly for ten years or so? I don't have enough technical knowledge to understand or fix the corruption. I just want to avoid it.

[PatrikLundell]

I suggest you take one of the corrupted saves on DFFD, disbands all squads, and creates new ones with only specific items for each member (in particular containing items from the corrupted categories, although you really should allocate all normal slots), and run it for at least 3 years, and you should also produce new equipment of kinds used (to try to force evaluations and see that they're not performed, basically). If that succeeds you've proven your concept. Personally I think it will fail, but I'll be happy to be proven wrong.

I'm not sure that this approach would eliminate corruption after it starts. My plan is to try it on a new fort where no soldier has ever been assigned non-specific equipment. I don't know much about the technical side, but wouldn't it demonstrate something if a fort like that ran smoothly for ten years or so? I don't have enough technical knowledge to understand or fix the corruption. I just want to avoid it.

I'm fairly sure it won't avoid corruption. Corruption happens because DF incorrectly considers some random data to be equipment, and thus adds it to whatever unmapped structures those are stored in. Storing potential equipment, should the player want to change assignments (or DF recalculate it) shouldn't in any way be affected by what the player has assigned currently (or, probably, whether the player has any militia at all, given my two cases of corruption without any units ever being sent out [except to exile]). The structures containing potential equipment would need to be maintained regardless.

Thus, what you would prove by successfully running a corrupted fortress with fully assigned equipment is that it would work as a work around (assuming it works, of course).  Running a fortress with fully assigned equipment from the start and not suffering any corruption only shows you didn't get corrupted, but not why. For that approach to provide any kind of indication of proof I'd require something like 10 fortresses each run for 40 years of heavy raiding, as some people are lucky and don't get corrupted, despite heavy raiding (my heavy raiding fortress was probably raiding for 20 years before it was corrupted, and that was before the patch that reduced the corruption rate).

Nobody but Toady has enough knowledge to understand equipment corruption, and he obviously struggles with it (or the fix would have fixed the problem, not just reduced the rate). If Toady was asked and provided enough info for the DFHack community to map the structures where the corruption happens, and optionally info on how data ends up added to those structures it might be possible for some people to use debuggers to catch and examine data entry, and others to detect it soon after the fact (by using a periodic script that scanned the structures e.g. every day). If the structures were known it might even be possible to remove corrupted entries via a script (Fixing the bug would obviously require not getting corrupted data into the system, but clearing it out before it causes a crash would be a work around until the causes have been eliminated).