Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[billw]

After struggling with stress issues in my latest fort and being unable to tell if my attempted remedies were having any effect, I decided to write a plugin that dumps stress levels so I can see their trend-lines over time.

If this is of interest to other people (perhaps something else that does this already exists, but I didn't find it), I can put in a PR to DFhack to get it added. It isn't trivial to use though, it requires knowledge of elasticsearch and kibana to get useful outputs from it (although I might make alternative outputs like Excel or something).

/edit:

Here is an example showing what happens when you burrow all your dwarves in the dining room with a dead body and lock the doors.
Note some of the data series terminating abruptly.

Spoiler (click to show/hide)

Also interestingly the only person not affected is my queen. In fact she is getting happier by the minute:

Spoiler (click to show/hide)

[Fleeting Frames]

What format is the dump, btw?
Something like .csv that digfort uses is pretty easy, just values separated by ;

[billw]

Currently it only outputs to elasticsearch. It gives live output and fairly good graphic, analysis and search functions. If there is something more appropriate and lighter weight I would be interested. This plugin is mostly useful to me because the output is live, dumping to a csv or whatever would not give a live output (unless there is some way to do this I am not aware of?). I could still do this if there is significant interest, but I wouldn't be using it personally.

[Fleeting Frames]

Aha, I see. I have not used the tools you use to display before; though I saw they were open-source with a brief search. Didn't catch that it was real-time updating, for instance.

There's also the third option of publishing separately like cavern keeper does. I can certainly see some potential for this family of tools; many times I've run checks for how much someone's attributes and gained experience changed in a time period, but that would be too much trouble to do by hand for everyone.

I can't say whether I'd use them any time soon, however.

And yeah, dumping to csv would be pretty static, afaik at least major tools like excel and libreoffice only read a snapshot of the file at the moment of opening.

[fortunawhisk]

Intriguing! I'd be interested to see how you did it (plugin vs lua vs ...?).

In terms of other tools, it depends on what part of your pipeline you want to replicate:
DwarfMonitor is probably the closest to the whole package, but doesn't exactly do what you're doing. http://dfhack.readthedocs.io/en/stable/docs/Plugins.html#id276
The dfhack script "repeat" gives you the ability to call a script on a consistent timed basis. https://dfhack.readthedocs.io/en/stable/docs/_auto/base.html#repeat
Dwarf Therapist gives you ability to see everything for everyone, but has no historical record or output options (afaik).

Couple of questions:
- Is the elasticsearch instance local or remote?
- How does the plugin pass multiple values? A single http push, multiple http pushes...?
- Any thoughts on how you'd convert timestamps to dwarf fortress timestamps? For instance, 1300-1330 -> Granite 01-28, Year 5.

[billw]

I also use the tools you mentioned so it's good to get some confirmation I didn't miss something obvious, thanks!
In fact I use the repeat script with this system for the timing at the moment. It just dumps the data one off to ES when called, and I schedule it with repeat to run once a day.
I used docker to setup ELK (https://hub.docker.com/r/sebp/elk/), I am running it on a second PC on the same network, but it could easily be remote or local I think, the data volume is very low. I use a single http session (I added libcurl to dfhack) and a separate push for each statistic. I couldn't see offhand how to push multiple docs in one go to ES, although I seem to remember that it is possible. I did think of trying to use df time AS the time stamps, however given what ES how kibana is designed it seemed like it would be going against the grain so I just use real time and added DF time as separate fields. It has worked okay so far. I might try it out though as I already wrote the code to do it.

//edit I might switch to using python, plotly and dash instead of ELK (https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503), as it is easier to distribute and is more appropriate for non "real world" metrics, i.e. not everything tied to a real world time stamp.

[fortunawhisk]

Here is a script I'm using to track stuff over time in fortress mode. It writes metrics to a CSV output, can track several different metrics, and is intended to be used with dfhack's repeat function.

Can monitor:
 - Total Experience in skills
 - Mental and Physical attributes
 - Personality facets
 - Stress
 - Focus
 - Needs
 - Beliefs (where they differ from the cultural norm)
 - All of the above

Output:
Writes to CSV (delimited by ","). Keeps a record of local timestamp, DF game timestamp (YY-MM-DD), a customizable "test tag", the dwarf name, and metrics. Metrics are written into a KeyValue Pair blob (Focus=90,Stress=3562).
Example output line:

Spoiler (click to show/hide)

2018-08-20_15:06:44,10-05-02,NoTag,`Hauler' Al†thkab(2135),Focus=90,Stress=3562,

Known Problems:
 - Output. The KVP style metric output makes it easy to know what the metrics are, somewhat challenging to graph.
 - I'm not sure about the best way to show needs actual strength.

Code: [Select]

-- Track changes to DF dorf stats
-- Monitor 1.2
-- Track changes by writing to a file
-- Fortress mode script called from DFHack
-- List functions from vjek's make-legendary
-- https://dfhack.readthedocs.io/en/stable/docs/_auto/base.html#make-legendary
-- LNP: [04412r02-x64]



local utils=require('utils')

validArgs = utils.invert({
 'help',
 'verbose',
 'outputfile',
 'targets',
 'watch',
 'tag',
 'noLog',
 'noTranslate'
})

local args = utils.processArgs({...}, validArgs)
local verbose = false
local translate = true
local noLog = false
local outputfile = "record.txt"
local targets = "all"
local watch = "skills"
local tag = "NoTag"
local watchTable = {"skills"}
local targetTable = {"all"}

local helpme = [===[

monitor.lua
=========
This script writes a record of dwarf stats to file. Stats are recorded in key-value pairs, so some spreadsheet kung-fu is required.
This script is intended to be used with the dfhack repeat function to record stats over time.

arguments:
    -help
        print this help message
-noLog
Don't write to the log file. Defaults to false (always write to log)
    -outputfile
        The filename to write to. Defaults to record.txt in the main DF directory
    -tag
        A settable string used to differentiate multiple collections runs. Examples: "Test1:NoTavern", "Test2:NoAlcohol"
    -targets
        A comma separated list of unit ids to monitor. Also takes all. Defaults to all, which is all citizens.
-noTranslate
Disable translation of names to English for display and write.
    -watch
        A comma separated list of stats you want to gather. Valid options are:
        skills: Default. The total experience in all skills
        attributes: The values for physical and mental attributes
        personality: The values for all personality facets
        stress: The unit stress level (integer)
        focus: The unit focus as a percentage: (current_focus/undistracted_focus)*100
        needs: The active need levels for a unit (1-10). 1 seems to be the highest
        beliefs: The active beliefs for a unit
relations: The relationship strength for non-family/deity relations
        all: All of the above. VERY VERBOSE.
    -verbose
        prints debug data to the dfhack console. Default is false

    Examples:
        Record the stress levels for all dwarves to the screen (once) with their untranslated names
        monitor -watch stress -verbose -translate false

        Record the stress levels for all dwarves (once)
        monitor -watch stress
   
        Record the skills of all dwarves to myfile.txt using the tag "Trial_1" (once)
        monitor -tag Trial_1 -outputfile myfile.txt
   
        Record the stress and focus levels for units 592 and 2135 every day
        repeat -name monitor -time 1 -timeUnits days -command [ "monitor -targets 592,2135 -watch focus,stress" ]
       

]===]

-- Handle help requests
if args.help then
    print(helpme)
    return
end

if ( args.verbose ) then
    verbose = true
end

if ( args.noTranslate ) then
    translate = false
end

if ( args.noLog ) then
    noLog = true
end

if ( args.targets ) then
    targetTable = {}
    -- Set the units to watch
    for i in (args.targets .. ","):gmatch("([^,]*),") do
        -- if verbose then print("Targeting: ".. i) end
        table.insert(targetTable, i)
    end
end

if ( args.tag ) then
    tag = string.gsub(args.tag, "%s+", "")
    tag = string.gsub(tag, ",", "")
end

if ( args.watch ) then
    watchTable = {}
    -- Set the groups to watch
    for i in (args.watch .. ","):gmatch("([^,]*),") do
        -- if verbose then print("Watching Group: ".. i) end   
        table.insert(watchTable, i)
    end
end

if( args.outputfile ) then
    outputfile = args.outputfile
end


function printTable(ts, df_ts, uname, tag, o)
    -- Now write all the data into an output file
    -- Expects a timestamp, a dwarf fortress timestamp, the dwarf's name, the run tag, and a data table
    io.write(ts .. "," .. df_ts .. "," .. tag .. "," .. uname .. ",")
    for k,v in ipairs(o) do
       
        io.write(string.gsub(v[2], "%s+", "").."="..v[3]..",")
    end
    io.write("\n")
end


function count_this(to_be_counted)
    local count = -1
    local var1 = ""
    while var1 ~= nil do
        count = count + 1
        var1 = (to_be_counted[count])
    end
    count=count-1
    return count
end

function PrintSkillList()
    local count_max = count_this(df.job_skill)
    local i
    for i=0, count_max do
        print("'"..df.job_skill.attrs[i].caption.."' "..df.job_skill[i].." Type: "..df.job_skill_class[df.job_skill.attrs[i].type])
    end

end

function PrintSkillClassList()
    local i
    local count_max = count_this(df.job_skill_class)
    for i=0, count_max do
        print(df.job_skill_class[i])
    end

end

function PrintNeedList()
    local i
    local count_max = count_this(df.need_type)
    for i=0, count_max do
        print(df.need_type[i])
    end

end

function PrintBeliefList(unit)
    local i
    local count_max = count_this(df.value_type)
    for i=0, count_max do
        print(df.value_type[i])
    end
end

function GetUnitSkills(unit, unitSkillTable)
    -- Get the unit's skills
    local i
    local tmpUnitSkillTable={}
    local count_max = count_this(df.job_skill)

    -- We need to add all the possible skills to a table with a default level and xp
    for i=0, count_max do
        -- Expected table structure is {id, skill caption, experience }
        local skill_name=string.lower(df.job_skill.attrs[i].caption)
        skill_name = string.gsub(skill_name, "^%l", string.upper)
        skill_name = string.gsub(skill_name, "%s+", "_")
        skill_name = string.gsub(skill_name, "[.-]", "_")
        skill_name = string.gsub(skill_name, "_%l", string.upper)
        skill_name = string.gsub(skill_name, "_", "")
        table.insert(tmpUnitSkillTable,{i,skill_name,0})
    end

    -- Unit skills are sparsely populated.
    -- We now need to loop over the unit skills and overwrite the default values with actual values
    -- DF starts indexes at zero, LUA starts tables indexes at 1
    for i=0, #unit.status.current_soul.skills-1 do
        tbl_skill = unit.status.current_soul.skills[i].id + 1
        skill_id = unit.status.current_soul.skills[i].id
        skill_lvl = unit.status.current_soul.skills[i].rating
        skill_exp = unit.status.current_soul.skills[i].experience
        local skill_name = tmpUnitSkillTable[tbl_skill][2]
        -- Experience formula courtesy of Maklak
        -- http://www.bay12forums.com/smf/index.php?topic=66525.msg3953394#msg3953394
        total_xp = math.floor(500*skill_lvl + 100*(skill_lvl*(skill_lvl-1))/2) + skill_exp
       
        if verbose then print (skill_id, " ", skill_lvl, " ", skill_exp, " ", total_xp ) end
       
        -- Overwrite the default skill table with unit specific skill levels
        tmpUnitSkillTable[tbl_skill] = {skill_id,skill_name,total_xp}
    end

    -- Replay table into target data structure.
    for i=1, #tmpUnitSkillTable do
       if verbose then print (i,tmpUnitSkillTable[i][1],tmpUnitSkillTable[i][2],tmpUnitSkillTable[i][3]) end
       table.insert(unitSkillTable,{i,tmpUnitSkillTable[i][2],tmpUnitSkillTable[i][3]})
    end
   
end

function GetUnitAttributes(unit, unitAttributeTable)
    -- Get the unit's mental and physical attributes
    -- Unit is expected to be unit id
    local i = 0
    for k,v in pairs(unit.status.current_soul.mental_attrs) do
        local attr_name = string.lower(tostring(k))
        attr_name = string.gsub(attr_name, "^%l", string.upper)
        attr_name = string.gsub(attr_name, "_%l", string.upper)
        attr_name = string.gsub(attr_name, "_", "")
        if verbose then print("Current value for "..attr_name.." is "..v.value) end
        table.insert(unitAttributeTable,{k,attr_name,v.value})
    end

    for k,v in pairs(unit.body.physical_attrs) do
        local attr_name = string.lower(tostring(k))
        attr_name = string.gsub(attr_name, "^%l", string.upper)
        attr_name = string.gsub(attr_name, "_%l", string.upper)
        attr_name = string.gsub(attr_name, "_", "")
        if verbose then print("Current value for "..attr_name.." is "..v.value) end
        table.insert(unitAttributeTable,{k,attr_name,v.value})
    end
end

function GetUnitPersonality(unit, unitPersonalityTable)
    -- Get the unit's personality. Apply temporary trait overlays
    -- Unit is expected to be unit id
    local i = 0
    local tmpUnitPersonalityTable= {}
    for k,v in pairs(unit.status.current_soul.personality.traits) do
        local trait_name = string.lower(tostring(k))
        trait_name = string.gsub(trait_name, "^%l", string.upper)
        trait_name = string.gsub(trait_name, "_%l", string.upper)
        trait_name = string.gsub(trait_name, "_", "")
        if verbose then print("Current value for " .. trait_name.." is "..v) end
        if(unit.status.current_soul.personality.temporary_trait_changes == nil) then
            -- Handle things with no trait changes
            if verbose then print("Final value for " .. trait_name.." is "..v) end
            table.insert(unitPersonalityTable,{i,trait_name,offset})
        else
            table.insert(tmpUnitPersonalityTable,{i,trait_name,v})
        end
        i = i + 1
    end
   
    if(unit.status.current_soul.personality.temporary_trait_changes == nil) then
        -- Exiting early
        print("Leaving personality w/o doing temporary_trait_changes")
        return   
    end
   
    i = 0
    for k,v in pairs(unit.status.current_soul.personality.temporary_trait_changes) do
        tbl_index = i + 1
        counter = tmpUnitPersonalityTable[tbl_index][1]
        trait_name = tmpUnitPersonalityTable[tbl_index][2]
        offset = tmpUnitPersonalityTable[tbl_index][3] + v
        if verbose then print("Offset:" .. trait_name .. " is " .. offset) end
   
        table.insert(unitPersonalityTable,{i,trait_name,offset})
        i = i + 1
    end
end   

function GetUnitStress(unit, unitStressTable)
    -- Get the unit's stress
    if verbose then print("Current value for Stress is "..unit.status.current_soul.personality.stress_level) end
    table.insert(unitStressTable,{1,"Stress",unit.status.current_soul.personality.stress_level})
end

function GetUnitFocus(unit, unitFocusTable)
    -- Get the unit focus as a percent
    local focus = math.floor((unit.status.current_soul.personality.current_focus/unit.status.current_soul.personality.undistracted_focus)*100)
    if verbose then print("Current value for Focus is "..focus .. "%:" .. unit.status.current_soul.personality.current_focus .. "/" .. unit.status.current_soul.personality.undistracted_focus) end
    table.insert(unitFocusTable,{1,"Focus",focus})
end

function GetUnitNeeds(unit, unitNeedsTable)
    -- Get the unit's needs
    local i
    local tmpUnitNeedsTable={}
    local count_max = count_this(df.need_type)
   
    -- We need to add all the possible needs to a table
    for i=0, count_max do
         -- Expected table structure is {id, need name, 0 }
        table.insert(tmpUnitNeedsTable,{i,df.need_type[i],0})
    end
   
    for i=0, #unit.status.current_soul.personality.needs-1 do
        tbl_index = unit.status.current_soul.personality.needs[i].id + 1
        needs_name = tmpUnitNeedsTable[tbl_index][2]
-- need_level is just a counter, not a multiplier
        needs_lvl = unit.status.current_soul.personality.needs[i].need_level
        needs_focus = unit.status.current_soul.personality.needs[i].focus_level
       
        -- This allows us to track the needs for multiple deities. Comparison will break if spelling is ever corrected
        if(needs_name == "PrayOrMedidate") then
            needs_name = needs_name .. "(" .. unit.status.current_soul.personality.needs[i].deity_id .. ")"
        end

       
        if verbose then print(tbl_index,needs_name,needs_lvl,needs_focus) end
        -- I'm hopeing that new needs will gracefully be added?
        table.insert(unitNeedsTable,{i,needs_name,needs_focus})
       
    end
end

function GetUnitBeliefs(unit, unitBeliefTable)
    -- Get the cultural beliefs, then overlay with the unit's personal beliefs.
    -- Courtesy of Atkana and the DFHack thread
    -- http://www.bay12forums.com/smf/index.php?topic=164123.msg7836773#msg7836773

    local j = 1
    local tmpUnitBeliefTable={}

    -- Check for the unit's culture in all the world's cultures.
    for i=0, #df.global.world.cultural_identities.all do
        if(unit.civ_id == df.global.world.cultural_identities.all[i].civ_id ) then
            for k,v in pairs(df.global.world.cultural_identities.all[i].values) do
              -- Break when we get to unnamed entries. Unknown why they are there.
              if(#tostring(k)<3) then break end
              if verbose then print("Cultural",j,k,v) end
              table.insert(tmpUnitBeliefTable,{j,k,v})
              j = j + 1
            end
            -- Stop after the first civ match
            break
        end
    end
   
    for i=0, #unit.status.current_soul.personality.values-1 do
        local tbl_index = unit.status.current_soul.personality.values[i].type + 1
        if verbose then print("Personal",i,tbl_index,tmpUnitBeliefTable[tbl_index][2],unit.status.current_soul.personality.values[i].strength) end
        tmpUnitBeliefTable[tbl_index] = {tbl_index,tmpUnitBeliefTable[tbl_index][2],unit.status.current_soul.personality.values[i].strength}
    end
   
    -- Replay table into target data structure.
    for i=1, #tmpUnitBeliefTable do
       local belief_name = string.lower(tmpUnitBeliefTable[i][2])
       belief_name = string.gsub(belief_name, "^%l", string.upper)
       belief_name = string.gsub(belief_name, "_%l", string.upper)
       belief_name = string.gsub(belief_name, "_", "")
       if verbose then print (i,tmpUnitBeliefTable[i][1],belief_name,tmpUnitBeliefTable[i][3]) end
       table.insert(unitBeliefTable,{i,belief_name,tmpUnitBeliefTable[i][3]})
    end
   
end

function GetUnitRelations(unit, unitRelationsTable)
-- Get the various relations for a unit. Goal here is to track relation strength
-- These are stored against the historical id, not the unit id. Some conversion is necessary.
-- Courtesy of PatrickLundell and the DFHack thread
-- http://www.bay12forums.com/smf/index.php?topic=172722.msg7896495#msg7896495

-- Initialize variables
local i = 0
local tmpUnitRelationsTable = {}

-- Convert unit id to historical id
local hf = df.historical_figure.find(unit.hist_figure_id)

-- confirm unit actually has relationships for listing

if(hf.info.relationships == nil ) then
return
-- Just stop
end


-- Get a list of unit's relation ids and strengths
-- Presumably, rank 0 means no relation
for i=0, #hf.info.relationships.list-1 do
if( hf.info.relationships.list[i].rank > 0) then
local r_histfig_id = hf.info.relationships.list[i].histfig_id
local r_local_id = df.historical_figure.find(hf.info.relationships.list[i].histfig_id).unit_id
local r_attitude = 0
local r_counter = 0
-- Does work: Testing for array length; Doesn't work: testing for next array element, testing for array, testing for exist
local attitude = #hf.info.relationships.list[i].attitude

if attitude >0 then
r_attitude = hf.info.relationships.list[i].attitude[0]
r_counter = hf.info.relationships.list[i].counter[0]
end

local r_rank = hf.info.relationships.list[i].rank
local myString = r_rank .. "(" .. r_attitude .. ")"

    -- Always convert historical ids back to unit ids
-- How should offsite/never arrived stuff work here?
-- Expected format here is #, local unit name, rank (attitude)

table.insert(unitRelationsTable,{i,dfhack.TranslateName(df.unit.find(r_local_id).name, translate),myString})
if verbose then print ("id:" .. i .. "|" .. dfhack.TranslateName(df.unit.find(r_local_id).name, translate) .. ": rank=" .. r_rank .. ", attitude=" .. r_attitude .. ", r_counter=" .. r_counter) end
end
end


end

function GetDFDate()
    -- Handle the formatting of the Dwarf Fortress date
    -- Thanks to Kurik Amudnil for the date stuff
    -- http://www.bay12forums.com/smf/index.php?PHPSESSID=669fc6cc7664043c4b34992a301abb0c&topic=91166.msg4247785#msg4247785
   
    -- Would it be useful to return a part of the DF date?
    -- local absTick = 1200*28*12*df.global.cur_year + df.global.cur_year_tick
    local dfYear = df.global.cur_year
    local dfMonth = math.floor((df.global.cur_year_tick / 33600) + 1)
    local dfDay = math.floor((df.global.cur_year_tick % 33600)/1200)+1
   
    if(dfMonth < 10) then
        dfMonth = "0"..dfMonth
    end
   
    if(dfDay < 10) then
        dfDay = "0"..dfDay
    end
   
    dfDateString = dfYear .. "-" .. dfMonth .. "-" .. dfDay
    return dfDateString
   
end

function SetTargets(targetTable)
    local tmpTargetTable = {}
    local citizenTable={}
   
    for k,u in ipairs(df.global.world.units.active) do
        if (dfhack.units.isCitizen(u)) then
            table.insert(citizenTable,u)
        end
    end
   
    for k, v in pairs(targetTable) do
        local check = targetTable[k]
        if verbose then print("Targeting: "..check) end
        if (string.lower(check) == string.lower("all")) then
            tmpTargetTable = citizenTable
            break -- Break out of the loop
        elseif tonumber(check) ~= nil then
            table.insert(tmpTargetTable, df.unit.find(check))
        else
            qerror('Error: Unhandled string. Was expecting a unit id or all')
        end
    end
   
    -- Overwrite target table with new targets
    targetTable = tmpTargetTable
    return targetTable
end

function SetWatchGroups(watchTable)
    local tmpWatchTable = {}

    for k, v in pairs(watchTable) do
        local check = watchTable[k]
        if verbose then print(check) end
        if string.lower(check) == string.lower('all') then
            -- Reset the table, only do all
            tmpWatchTable = {}
            table.insert(tmpWatchTable, "all")
            break -- Break out of the loop
        elseif ( (check == "skills") or
                 (check == "attributes") or
                 (check == "personality") or
                 (check == "stress") or
                 (check == "focus") or
                 (check == "needs") or
                 (check == "beliefs") or
(check == "relations") ) then
            table.insert(tmpWatchTable, check)
        else
            qerror('Error: Unhandled -watch string, try -help')
        end
    end
   
    -- Overwrite target table with new targets
    watchTable = tmpWatchTable
    return watchTable
end
       
-- Let's do this

-- Opens a file in append mode
file = io.open(outputfile, "a")
io.output(file)

-- Get the current time
local runTimestamp = os.date("%Y-%m-%d_%H:%M:%S")

-- Get the DF date
local dfDay = GetDFDate()

-- Set the units to watch
targetTable = SetTargets(targetTable)

-- Set the things to watch
watchTable = SetWatchGroups(watchTable)

-- Get list of valid targets(citizens)
for k,u in ipairs(targetTable) do
    if verbose then print("Collecting stats for: ", dfhack.TranslateName(dfhack.units.getVisibleName(u), translate)) end
    local uKVPTable={}
    local unitName = dfhack.TranslateName(dfhack.units.getVisibleName(u), translate) .."(".. u.id .. ")"
       
    for x,watch in pairs(watchTable) do
        if verbose then print(watch) end
        if(watch == "skills") then
            GetUnitSkills(u, uKVPTable)
        end
       
        if(watch == "attributes") then
            GetUnitAttributes(u, uKVPTable)
        end
       
        if(watch == "personality") then
            GetUnitPersonality(u, uKVPTable)
        end
       
        if(watch == "stress") then
            GetUnitStress(u, uKVPTable)
        end

        if(watch == "focus") then
            GetUnitFocus(u, uKVPTable)
        end
       
        if(watch == "needs") then
            GetUnitNeeds(u, uKVPTable)
        end

        if(watch == "beliefs") then
            GetUnitBeliefs(u, uKVPTable)
        end

        if(watch == "relations") then
            GetUnitRelations(u, uKVPTable)
        end
       
        if(watch == "all") then
            GetUnitSkills(u, uKVPTable)
            GetUnitAttributes(u, uKVPTable)
            GetUnitPersonality(u, uKVPTable)
            GetUnitStress(u, uKVPTable)
            GetUnitFocus(u, uKVPTable)
            GetUnitNeeds(u, uKVPTable)
            GetUnitBeliefs(u, uKVPTable)
    GetUnitRelations(u, uKVPTable)
        end
       
    end
if noLog then
-- nothing. just do nothing
else
printTable(runTimestamp, dfDay, unitName, tag, uKVPTable)
end
end

io.close(file)

Edit: All belief values are now available.
Edit 20190324: Fixes and changes
-- I had focus_level multiplied by need_level. This is wrong, it should just increment by need_level.
-- Translating names is now globally consistent via noTranslate option
-- You can now disable logging via noLog option
-- Script now tracks relationship strength for acquaintances (rank).

[Fleeting Frames]

I personally 'solved' that KVP problem by inputting on first line what the keys are in my orbituary assessment script.

Would be hard to read with dozen different keys in a text file, but easy in a spreadsheet program. Though you'd have to pre-sort it per dwarf.

Also, thx, I was contemplating writing something for experience myself.

[billw]

I'm in the process of upgrading my Elasticsearch and Kibana system as I mentioned:
- libcurl is now exposed as lua functions in dfhack so scripts can send arbitrary objects over REST (https://github.com/billw2012/dfhack/tree/libcurl)
- a python based server that can accept the json objects, store them to a db (anything pandas supports is easy, currently I'm using an xlsx file with a separate sheet for each metric) (https://github.com/billw2012/df_analytics)
- Dash+plotly for live visualization, I didn't write this bit yet, probably start it tomorrow

I think you could easily redirect your script outputs to my server and get live output if you are interested.

[Meph]

Could this Info, maybe not all, but part of it, displayed ingame in a gui? Like dwarfmonitor?

[billw]

Could this Info, maybe not all, but part of it, displayed ingame in a gui? Like dwarfmonitor?

Sure. I normally play in windowed mode with about 4 other windows open across two monitors, so for me it is preferable to have it out of the game. Also the lack of decent GUI in game is going to make graphs and plots rather limited.

/quick edit: for me, the point of this system is to put the data into proper analytics tools such as those that Python provides, so I can do easy and quick iteration on computation, comparison and display.

[Fleeting Frames]

I suppose one could theoretically do thin overlapping lines and such with a compiled plugin; such as how stonesense completely redraws everything.
(Doing it in lua...Maybe by dynamic generation of graphics tiles then assigned to a dummy monster, assuming you could assign arbitrary data.)

You'd have to reimplement spreadsheet software, of course. Bit grand, that *pokes libreoffice-openoffice relationship with a stick*

[fortunawhisk]

@Fleeting Frames: My goal with the KVP mess was to make sure I could compare any single run with any other, regardless of metrics collected, order specified, etc. It works, but I spend a lot of time on spreadsheet layout and formulas now...

@Meph: Like they said, you could probably store the historical data inside DF somehow and create a chart or table for it. My opinion is that Dwarf Therapist is probably the best fit tool, but stapling a db and graphing package to it probably isn't anyone's idea of fun.

@billw: I'm aiming to not have to install vm's/containers to make this useful (to me or anyone else), but I'm really tired of spreadsheets at this point.

[lethosor]

I suppose one could theoretically do thin overlapping lines and such with a compiled plugin; such as how stonesense completely redraws everything.
(Doing it in lua...Maybe by dynamic generation of graphics tiles then assigned to a dummy monster, assuming you could assign arbitrary data.)

You'd have to reimplement spreadsheet software, of course. Bit grand, that *pokes libreoffice-openoffice relationship with a stick*

Stonesense draws to a separate window using Allegro by default (it doesn't go through DF's graphics system at all, or even SDL). The overlay copies the contents of that window back to the DF window, but is also buggy (not solely due to that, though), and would probably be overkill for a basic UI.

While I'm sure that basic information could be displayed in-game somehow, the idea here seems to be exporting data for other analysis tools, so I agree that an in-game UI might not be necessary.

[billw]

@fortunawhisk Yeah I want the same thing, that is a major reason why I am switching away from Elasticsearch and Kibana to Python and a web browser, which could just be made into a single installer or zip file for anyone to use.

I'm hoping by using a very popular language (possibly THE most popular one for data analysis) it will mean that more other people will be interested in writing their own frontends/graphs/etc. for this system and then sharing them.