Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Nilsolm]

Originally posted on reddit, I wanted to get some input on this from here as well, in case I am misinterpreting something.

It seems that the job distribution of histfigs is completely buggered in the current version. Specifically, there are way too many animal caretakers and herbalists in the histfig pool. The cause of this appears to be that something is wrong with how jobs are selected by people in elven settlements. If you look at any forest retreat in legends mode, you'll see that people only ever take those two jobs (apart from the odd performer, scout and scholar). This doesn't seem right of course, because people should be able to have more varied professions according to the raws:

Code: [Select]

[SCOUT]
[SCHOLAR:PHILOSOPHER]
[SCHOLAR:ASTRONOMER]
[SCHOLAR:NATURALIST]
[SCHOLAR:GEOGRAPHER]
[PERMITTED_JOB:BOWYER]
[PERMITTED_JOB:ANIMAL_CARETAKER]
[PERMITTED_JOB:WOODCRAFTER]
[PERMITTED_JOB:WEAVER]
[PERMITTED_JOB:CLOTHIER]
[PERMITTED_JOB:HERBALIST]
[PERMITTED_JOB:TRADER]

So I tried have a closer look at some relevant data to try and work out what might be happening. I wrote two scripts (see below) to extract what data is needed and to make a few hideous looking plots. Specifically, what is interesting is:

• what professions are the most frequent (I removed soldiers and some other non-relevant jobs from the list here), and
• what new jobs people take most frequently. Whenever people become adults or have a change of heart, there is a change_hf_job-type event with a new_job value.

Then I had a look at four worlds: two with the default raws and two with modified raws. For the latter, I just removed animal caretaking and herbalism as permitted jobs from the elves.

Default raws, 250 years of history

Histfig professions: https://i.imgur.com/6qocGqb.png
New_job values: https://i.imgur.com/oGGlVnG.png

As you can see, something isn't right. There are way too many animal caretakers and herbalists. Those jobs are way more common than they should be. Most of this seems to stem from people taking those jobs in elven settlements.

Default raws, 500 years of history

Histfig professions: https://i.imgur.com/wECYyCj.png
New_job values: https://i.imgur.com/o4pIfL5.png

Same situation, except it seems to be even more extreme. The longer the world history, the more the job distribution seems to skew towards those two professions.

Modified raws, 250 years of history

Histfig professions: https://i.imgur.com/ZN2AvuU.png
New_job values: https://i.imgur.com/tD6abIU.png

Now, removing those two lines from the raws seems to solve the problem of animal caretaking and herbalism being overrepresented. However, there is something else not right here. Look at the discrepancy for the remaining permitted jobs. Clothier, weaver and bowyer are very low on the list of new_job values, and woodcrafter does not even seem to appear once. Despite that, those professions seem unusually common.

Modified raws, 510 years of history

Histfig professions: https://i.imgur.com/uvSRm93.png
New_job values: https://i.imgur.com/DZfpByO.png

Again, same situation. People don't really become clothiers etc. that often, but those jobs are still some of the most frequent ones.

Conclusion

Something seems to be wrong with the elves. I don't really know why though. I don't remember this being an issue before 47.01 (although I still need to look at some 44.12 data). I combed through the devlogs, but I didn't see anything obvious that might be causing this. My guess is that it's related to the guilds.

Scripts

Here are the scripts I used, in case someone wants to give it a try as well.

DFHack script to export the relevant data:

Spoiler: exportjobdata.lua (click to show/hide)

Code: [Select]

local gui = require 'gui'
local script = require 'gui.script'
local args = {...}
local vs = dfhack.gui.getCurViewscreen()

function move_back_to_main_folder()
    return dfhack.filesystem.restore_cwd()
end

local folder_name = "job_data"
dfhack.filesystem.mkdir(folder_name)
-- Go to save folder, returns true if successfully
function move_to_save_folder()
    if move_back_to_main_folder() then
        return dfhack.filesystem.chdir(folder_name)
    end
    return false
end

function progress_ipairs(vector, desc, interval)
    desc = desc or 'item'
    interval = interval or 10000
    local cb = ipairs(vector)
    return function(vector, k, ...)
        if k and #vector >= interval and (k % interval == 0 or k == #vector - 1) then
            print(('        %s %i/%i (%0.f%%)'):format(desc, k, #vector, k * 100 / #vector))
        end
        return cb(vector, k)
    end, vector, nil
end

function escape_xml(str)
    return str:gsub('&', '&amp;'):gsub('<', '&lt;'):gsub('>', '&gt;')
end

local df_enums = {} --as:df
setmetatable(df_enums, {
    __index = function(self, enum)
        if not df[enum] or df[enum]._kind ~= 'enum-type' then
            error('invalid enum: ' .. enum)
        end
        local t = {}
        setmetatable(t, {
            __index = function(self, k)
                return df[enum][k] or 'unknown ' .. k
            end
        })
        return t
    end,
    __newindex = function() error('read-only') end
})

if not move_to_save_folder() then
    qerror('Could not move into the save folder.')
end

local filename = "job_data.xml"
local file = io.open(filename, 'w')

move_back_to_main_folder()
if not file then
    qerror("could not open file: " .. filename)
end

file:write("<?xml version=\"1.0\" encoding='UTF-8'?>\n")
file:write("<jobs>\n")
for hfK, hfV in progress_ipairs(df.global.world.history.figures, 'historical figure') do
    file:write("<historical_figure>\n")
    if hfV.race >= 0 then
        file:write("\t<race>"..escape_xml(dfhack.df2utf(df.creature_raw.find(hfV.race).name[0])).."</race>\n")
    end
    file:write("\t<profession>"..df_enums.profession[hfV.profession]:lower().."</profession>\n")
    file:write("</historical_figure>\n")
end

for ID, event in progress_ipairs(df.global.world.history.events, 'event') do
    if df.history_event_change_hf_jobst:is_instance(event) then
        file:write("<job_change_event>\n")
        for k,v in pairs(event) do
            if df.history_event_change_hf_jobst:is_instance(event) and (k == "new_job" or k == "old_job") then
                file:write("\t\t<"..k..">"..df_enums.profession[v]:lower().."</"..k..">\n")
            end
        end
        file:write("</job_change_event>\n")
    end
end
file:write("</jobs>")
file:close()


This is based on the exportlegends script. I basically just ripped out the parts that I needed. Call it from legends mode and it creates a job_data.xml in the main DF folder.

Python script to parse the xml and plot the data:

Spoiler: plotdata.py (click to show/hide)

Code: [Select]

import xml.etree.ElementTree as ET
import matplotlib.pyplot as plt

# Parse xml
context = ET.iterparse("job_data.xml", events=("start", "end"))
context = iter(context)
ev, root = next(context)

# Find histfig professions and new_job values
professions = []
newjobs = []

for ev, el in context:
    if ev == "start" and el.tag == "profession":
        if el.text is not None:
            professions.append(el.text.lower())
        root.clear()
    elif ev == 'start' and el.tag == 'new_job':
        if el.text is not None:
            newjobs.append(el.text)
        root.clear()

# Remove all the fluff
bollocks = ["recruit", "pikeman", "master_pikeman", "trained_war", "trained_hunter", "blowgunman", "master_blowgunman", "none", "drunk", "standard", "baby", "child", "swordsman", "axeman", "maceman", "hammerman", "spearman", "lasher", "pikeman", "crossbowman", "bowman", "wrestler", "master_swordsman", "master_axeman", "master_maceman", "master_hammerman", "master_spearman", "master_lasher", "master_pikeman", "master_crossbowman", "master_bowman", "master_wrestler"]
professions_filtered = [x for x in professions if x not in bollocks]
newjobs_filtered = [x for x in newjobs if x not in bollocks]

# Get jobs and job count for professions
count_prof = []
jobs_prof = []

for job in professions_filtered:
    if job not in jobs_prof:
        jobcount = professions_filtered.count(job)
        count_prof.append(jobcount)
        jobs_prof.append(job)

# Get jobs and job count for new_jobs
count_new = []
jobs_new = []

for job in newjobs_filtered:
    if job not in jobs_new:
        jobcount = newjobs_filtered.count(job)
        count_new.append(jobcount)
        jobs_new.append(job)

# Sort by job count
jobs_prof_sorted = [x for _, x in sorted(zip(count_prof, jobs_prof))]
count_prof_sorted = [x for x, _ in sorted(zip(count_prof, jobs_prof))]

jobs_new_sorted = [x for _, x in sorted(zip(count_new, jobs_new))]
count_new_sorted = [x for x, _ in sorted(zip(count_new, jobs_new))]

# Normalise count values
count_prof_norm = [float(i)/max(count_prof_sorted) for i in count_prof_sorted]
count_new_norm = [float(i)/max(count_new_sorted) for i in count_new_sorted]

# Categorise professions
miners = ["miner"]
woodworkers = ["woodworker", "bowyer", "carpenter", "woodcutter"]
stoneworkers = ["stoneworker", "engraver", "mason"]
rangers = ["hunter", "animal_caretaker", "animal_dissector", "animal_trainer", "trapper", "ranger"]
doctors = ["doctor"]
farmers = ["planter", "beekeeper", "brewer", "butcher", "cheese_maker", "cook", "dyer", "gelder", "farmer", "herbalist", "lye_maker", "milker", "miller", "potash_maker", "presser", "shearer", "soap_maker", "spinner", "tanner", "thresher", "wood_burner"]
fishers = ["fisherman", "fishery_worker", "fish_dissector", "fish_cleaner"]
metalsmiths = ["armorer", "furnace_operator", "metalcrafter", "weaponsmith", "blacksmith", "metalsmith"]
jewelers = ["jeweler", "gem_cutter", "gem_setter"]
crafters = ["craftsman", "woodcrafter", "stonecrafter", "leatherworker", "bone_carver", "weaver", "clothier", "glassmaker", "strand_extractor", "papermaker", "wax_worker", "potter", "bookbinder"]
engineers = ["engineer", "mechanic", "siege_engineer", "siege_operator", "pump_operator"]

# Assign colours based on profession (default DF colour scheme used)
colours_prof = [
    "#C0C0C0" if y in miners
    else "#FFFF00" if y in woodworkers
    else "#000000" if y in stoneworkers
    else "#008000" if y in rangers
    else "#800080" if y in doctors
    else "#808000" if y in farmers
    else "#000080" if y in fishers
    else "#808080" if y in metalsmiths
    else "#00FF00" if y in jewelers
    else "#0000FF" if y in crafters
    else "#FF0000" if y in engineers
    else "#800080"
    for y in jobs_prof_sorted
]

colours_new = [
    "#C0C0C0" if y in miners
    else "#FFFF00" if y in woodworkers
    else "#000000" if y in stoneworkers
    else "#008000" if y in rangers
    else "#800080" if y in doctors
    else "#808000" if y in farmers
    else "#000080" if y in fishers
    else "#808080" if y in metalsmiths
    else "#00FF00" if y in jewelers
    else "#0000FF" if y in crafters
    else "#FF0000" if y in engineers
    else "#800080"
    for y in jobs_new_sorted
]

# Plot data
fig, ax = plt.subplots(figsize=(18,12))

ax.barh(jobs_prof_sorted, count_prof_norm, color=colours_prof)
ax.set_title("Histfig professions")
ax.grid(which="both", axis="x")
fig.savefig("jobdistribution.png")

fig2, ax2 = plt.subplots(figsize=(18,12))

ax2.barh(jobs_new_sorted, count_new_norm, color=colours_new)
ax2.set_title("New_job values of change_hf_job events")
ax2.grid(which="both", axis="x")
fig2.savefig("newjobs.png")


Run this in the folder where job_data.xml is located. Requires matplotlib. It created two plots: one for the job distribution and one for the frequency of new_job values.

[Mobbstar]

For reference, this had also been touched upon in the Future Of The Fortress thread. Toady suspects elves dominate performance skills and thus lure dwarves and/or histfigs into elven society, thereby setting off the skew you've described here.

Do professions have any inherit bias for/against?

[Nilsolm]

I know, that discussion is partly what prompted me to look into this in the first place.

I'm not sure about that performer explanation though. Based on what I've seen here, dwarves flocking to elven sites for one reason or the other is part of the problem, but it doesn't seem to be the whole story. There is something that makes everyone predisposed to taking up animal caretaking and herbalism over all other professions. Taking away those two from the elves still appears to lead to elven professions being disproportionately common, but they don't completely dominate the histfig pool anymore. So there is probably some kind of bias involved, but I don't quite understand how.

My guess is that job selection in elven sites is bugged somehow. One way to confirm this would be to completely change up the elves' permitted jobs and see how that affects things. I'll try that later.

[DwarfStar]

It sounds like there is a self-reinforcing effect, that causes any popular profession to become more popular with time. I would guess that existing workers somehow can cause other citizens to take the same profession. Maybe lineage? (If animal caretakers’ kids are more likely to become animal caretakers.) I guess there needs to be an “economics” term to increase demand for less popular jobs.

[PatrikLundell]

If guilds were modeled in settlements other than dwarven ones, establishment of guilds would train kids in those professions (it happens in your own fortress: kids mooding in forges). However, I don't think guilds do affect things elsewhere.

[Starver]

In general, though, quite a lot of our own ancestors were taking care of animals or harvesting plants.

Not that it invalidates the assessment, or complaint. As a world develops, perhaps metalworkers should beget more metalworkers, frexample, taking from the pool of excess and uninclined-to-inherit-the-agricultural-profession offspring - or some sort of dominant/recessive geneology-of-career-tendency to skew things from cross-lifestyle family pairings.

Supply/demand is the ultimate arbiter. Though surely the intention is that you're not going to get anywhere as many high-skill armourers immigrating as farmers, and for the current (apparent) likelihood to hold true there must still be an agrarian-bias to to the greater world (obv. taking into account who is likely to up sticks from their original settlement, and who may not). A different mix in town-sized places, and yes I could see armourers as most prolific in any place with castle qualities.  Entertainers might fill in the slack (a la Civilization's "neither scientist (research) nor tax-collector (gold)" basic trimvirate of resource-farming where happiness is the thing you are forced to at least partially make your 'production' aim) or have true value (culture, plus information?) but I find that to be the hardest justification.

Except with elves. Being above it all (literally, as well as figurativel) I can imagine their renaissance being much in advance of the humans or dwarves. A glorified pastoral/poetic mix (give or take extreme environmentalist/cannabilistic attitudes when such harmonies are disturbed in front of them) might be acceptable. Leading only to questions as to how these may (cult-like) attract more followers of such an inclination than a dwarven attitude of industrial perfection (in all things, including arable farming). So tweaks and overhauls might be useful, but not necessarily upon the initial premise.


(How a player skews their own embark, or if an Adventurer decides to go off and kill everyy cheesemaker in the land for whatever reason, is of course still the choice of the one in charge of the game. I'm trying to address here the abstracted NPC issues, and then mostly in the zero-player proportion of the game, before particularly forceful gaming might propogate cultural shifts all across the post worldgen landscape. "Send me all your best architects, upon pain of pain!", perhaps, as a demand of tribute once the ability is there to back up that 'promise'.)

[Nilsolm]

It sounds like there is a self-reinforcing effect, that causes any popular profession to become more popular with time. I would guess that existing workers somehow can cause other citizens to take the same profession. Maybe lineage? (If animal caretakers’ kids are more likely to become animal caretakers.) I guess there needs to be an “economics” term to increase demand for less popular jobs.

That would be an interesting factor to look at. I briefly skimmed one of the legends.xml files I used, but I didn't see any obvious connection between lineage and profession. I could probably get some actual data on this, but I'd have to modify the scripts accordingly.

If guilds were modeled in settlements other than dwarven ones, establishment of guilds would train kids in those professions (it happens in your own fortress: kids mooding in forges). However, I don't think guilds do affect things elsewhere.

 
You're right. Guilds apparently do not exist anywhere outside of dwarven settlements as of now, except for the occasional guildhall built by dwarves in non-dwarven sites they conquered.

In general, though, quite a lot of our own ancestors were taking care of animals or harvesting plants.

Not that it invalidates the assessment, or complaint. As a world develops, perhaps metalworkers should beget more metalworkers, frexample, taking from the pool of excess and uninclined-to-inherit-the-agricultural-profession offspring - or some sort of dominant/recessive geneology-of-career-tendency to skew things from cross-lifestyle family pairings.

snip

Certainly, some bias in the job distribution would be reasonable, considering the real-world time period the game is based on. This is more about those two specific professions being overrepresented to such a degree that I'm fairly certain it's not intended. And getting swamped by animal caretaker/herbalist migrants seems to be a common complaint nowadays.

In fact, I got some data from v44.12 for reference. Default raws, 250 years of history:

Histfig professions: https://i.imgur.com/SZBLv8F.png
New_job values: https://i.imgur.com/2uFnSlQ.png

Herbalists and animal caretakers are near the top again, but so are all the other elven jobs this time.  It seems fairly reasonable overall and much more in line with what you get in 47.05 by modifying the raws.

[DwarfStar]

(Maybe you're already doing this but) It would be interesting to hear what distribution you get with v44.12 at 500 years. If we're right that something only in the new version is causing existing workers to "breed", then we should expect the distribution to not have shifted significantly in any direction.

[Nilsolm]

Here is v44.12 world at 550 years:

Job distribution: https://i.imgur.com/HLYlfKM.png
New_job values: https://i.imgur.com/eIgBCHl.png

Overall very similar to what you get with 250 years. Those extra 300 years don't seem to make a difference.

Not really sure where to go from here though. Maybe look at all the minor releases since 47.01? I skimmed through all the changelogs, but nothing in there seemed particularly relevant. Or I could change up elven permitted jobs completely and see what happens. Maybe that'll help narrow things down a bit.