Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Scruiser]

From this and your numbers it seems that there is a start around 1000-1500 credits for a dabbling researcher

I wonder if Toady dropped a zero by mistake when describing how much research credit is awarded. (he said the base level was 100, maybe he meant 1000?).  I was getting absurdly long estimates when calculating how long it should take unskilled researchers, I think they look long, but more reasonable with 1000 as the base level for dabbling researchers.

If we get enough data to make a rough table, we could identify key break points (skill level required to get a research dice with the first research credits awarded, etc.).  For players trying to optimize and min max embark point spending, this could prove very valuable.

(didn't see how much time pondering did it take)

I was checking every 2-4 in game days in order to verify my estimate/calculation of an average of 58.54 times pondering to get research credit.  It probably isn't worth our time to further confirm this given how tedious it is to check for research credits and times pondering every single time.

Edit: Optics Engineer is up to 5600, an increase of 1400.  Still at competent skill level.  Somewhere between 55 and 60 times pondering.

Edit 2: Astronomer somehow reset back to 0 credits?  Not sure if it is a glitch in the dfHack script, or a bug in the research credit system.  If it's a bug... that's pretty severe to periodically lose all research credits.  Or can skill rolls come up negative on critical failures?

Edit 3: Proficient Chemist is up to 9500, an increase of 2300 credits.    Somewhere between 59 and 65 times pondering.

Edit 4: Proficient Mathematician is up to 9700 credits, an increase of 2700 credits.   62-70 times pondering.  I am starting to wonder if the first gain in credits is systematically bigger?  A bonus for the first time getting credits on a topic?

[Bumber]

Edit 2: Astronomer somehow reset back to 0 credits?  Not sure if it is a glitch in the dfHack script, or a bug in the research credit system.  If it's a bug... that's pretty severe to periodically lose all research credits.  Or can skill rolls come up negative on critical failures?

Did they switch topics?

[anewaname]

...
if I did everything right, this shouldn't error out. So, I think what we need to do next is have as many people run this script locally, share the csv files with us and then use something like python or something to add all the numbers together and then figure out the highest 10 correlating topics for each topic? I don't have a lot of experience with data crunching, so I am kinda wandering around blindly here...
...

I would go with a second lua script to process the data. This script will find the available extracts, but the code needs to be added that will process each file. Save it as a lua file in the hack/scripts folder and run it in dfhack. You do not need to load a fort to run it.

Spoiler (click to show/hide)

Code: [Select]

-- rough flow of process...
--  begin logging
--  remove old sum file because it will be replaced
--  find list of *-topic-correlation.csv files (in primary df folder)
--  for each file in list, attempt to read in the contents and accumulate the data, logging some info regarding how it went
--  close log


--start logging
local log = io.open("sum-topic-correlation.log", 'w')
log:write (os.date().."Starting process.\n")

--remove old sum file if exists
local out= "sum-topic-correlation.csv"
local b=os.remove (out); --attempt to delete old sum file
if b==nil then print "found no sum file" else print "found and deleted old sum file\n" end; -- this may not recognize if the sum file is locked by another app

-- obtain list of extracts from df folder
local filelist = io.popen("dir *-topic-correlation.csv")
local s=filelist:read("*l"); --get first filename
  if s==nil then log:write (os.date().." Found no *-topic-correlation.csv files.\n") end; --logs it if there are no files found
while s~=nil do --while there are files to process, attempt to process...
    print ("Processing "..s); log:write (os.date().." Processing "..s.."\n")

    -- code needed here
    print ("   code needed to open file, read contents, merge data, close file, and log what happened")
 
 
   
    s=filelist:read("*l"); --get next filename
end

-- dump accumulation into sum file
local sumfile = io.open ("sum-topic-correlation.csv", 'w')
    -- code needed here
sumfile:write ("code needed here to populate this file with accumulated data...")
sumfile:close()   
   

--end logging
log:write (os.date().."Ending process.\n")
log:close()


Regarding the data collection, the current extract has already starting crunching data.

I would suggest changing the extract to contain a list of historic figures and some of their data. The main reason for this is that large files can be processed at a blistering rate and unused data can be tossed aside, but if you are wondering if the mechanics or architect skill might influence what secret knowledge is gained, it takes time to ask players to extract more data. If each line of the extract has something like {histfig unit id}{expanded secrets.knowledge vector}{expanded secrets.whatever vector}{skills vector}, then you have the data upfront. Adding knowledge and skill levels also would allow further comparisons.

Just my point of view on data crunching...

[hun]

Regarding the practical implications for a science fort, do all the mechanics imply that it is much better to make separate libraries for different academic areas? Do the scholars lead the discussion only with the scholars of the same library?

If it is the case, then the separated libraries has two advantages: 
1) there would be discussions only regarding the specialty of the library, thus a lot of focused skill boost rather than situations when a skilled mathematician gaining xp for useless for him geography.
2) it can be the best way to train scholars from zero xp to a specific academic skill. Sure, initially a newbie starts working on a random topic, but there are gonna be a lot of focused discussions, so after he finishes working on the first topic, he is likely to have a higher skill in library's academic area and would start working only on related topics.

[Scruiser]

Regarding the practical implications for a science fort, do all the mechanics imply that it is much better to make separate libraries for different academic areas?

You are correct that scholars only discuss with scholar sharing their library.  In addition, discussing only takes one day while pondering takes two.  So I think you are right, specialty libraries focused on separate skill areas makes sense.  I would add that you should give skilled researchers their own separate libraries, and then give them an unskilled dwarf to work with.  The unskilled dwarf will gain skill from discussing with them, discussing awards more XP than pondering (10-30xp for discussing versus 5 xp for pondering according to therahedwig's tests).  Also, even if the skilled dwarf wastes on average half their time discussing the unskilled dwarf's topic instead of having the unskilled dwarf discuss their topic that will still not add any time to getting research done because discussing only takes 1 day while pondering takes 2.

For training scholars from 0xp, note that several topics actually use skills that have practical use and that can be trained other ways, in particular a bunch of engineering topics use mechanics, many medical topics using surgery or diagnostician, one topic uses organizer, and a few topics use record keeper.  So you could have mechanics churn out a bunch of gears to grind their skill before setting them to do research.  And you could rotate out noble assignment of manager and chief medical dwarf to scholars.

Did they switch topics?

They were pondering the same thing (path of the moon I think?  I can check tonight) according to the in game labor description and the dfHack script output looked the same as before they earned research credit.

Edit: checking through again tonight... I've got a dabbling logician that got 800 research credit after 71-72 times pondering.  So the base level of research credit isn't 1000, it's just that the skill roll for even unskilled dwarfs adds a much larger amount than the base value of 100.  Also, I checked and my astronomer is still pondering the same thing and still at 0 research credits, so somehow research credit can go down or dfHack is reading it wrong...

Edit 2: Proficient Chemist is up to 15200 from 9500, an increase of 5700, after 50-60 times pondering.  Skilled Optics engineer is up to 8400 from 5600, an increase of 2800.

Edit 3: Competent Astronomer, that had previously been set back to 0 by some glitch or undocumented feature, is now at 3100 (52-60 times pondering).

Edit 4: Proficient Mathematician, up to 15300 from 9700, an increase of 5600, after 60-70 times pondering.

Edit 5: Competent Astronomer is up to 5900 from 3100, an increase of 2800.  Novice Logician is up to 2500 from 800, an increase of 1700.

Edit 6: Skilled Optics Engineer up to 12100 from 8400, an increase of 3700. 

Edit 7: Talented Chemist is down to 0 from 15200, still pondering the same topic (Hardness).  I think it is important determining if this is a bug in the script or a bug in the game or if Toady missed describing something important.

Edit 8: Talented Mathematician up to 19200 from 15300, an increase of 3900.

[hun]

For training scholars from 0xp, note that several topics actually use skills that have a practical use and that can be trained other ways, in particular, a bunch of engineering topics uses mechanics, many medical topics using surgery or diagnostician, one topic uses organizer, and a few topics use record keeper. 

Yes, you are right, engineering and medical school should be really easy to set up. I actually forgot about record keeper and organizer, it is neat to remember.
But the main challenge is to train mathematicians or chemists or other "pure" scientists.  And that is where focused discussions can help.

Btw, instead of dumping all the observations about scholars getting credits in this topic, I will just set up a table and will be collecting the data for a while, and upload it later. I think you could also make a summary of your observations, it would allow for easier analysis later.
I will be recording an academic topic, scholar's skill level, and the progress of the credits (record current progress number every time the credits increase or decrease).

EDIT: my chemist ponders Combustible Materials, but this topic is not displayed in learning. It might be just a rare bug of the script or dfhack. The number of times pondered seems to reset normally.

[Scruiser]

I also have a topic being pondered that doesn't show up correctly in the dfHack script: Shadow Clocks.  Just a blank spot for it, but the number of times pondered seems to go up and reset correctly for it.

I'll start compiling my results into a table as well.  What do you think should go in a table?  I think at minimum skill level (of the relevant skill) and research credits gained are interesting and useful.  I think a useful end goal would be the average number of credits gained across various levels of skill.  Number of times pondered is tedious to track, since you have to get it right before it succeeds and resets to zero, and I don't think it is a useful number since it is the same across skill levels and my initial calculation of 58.5 times on average seems to be accurate enough.

[therahedwig]

I also have a topic being pondered that doesn't show up correctly in the dfHack script: Shadow Clocks.  Just a blank spot for it, but the number of times pondered seems to go up and reset correctly for it.

Yes, this is because I am an idiot; the first entry from a list of topics doesn't show, and I haven't yet figured out why, but it is proly something stupid if my past programming experience has taught me anything.

For the record, when I was messing about to get the skill info, a legendary critical thinker could get 25K research credit one cycle and 3K the next, so it is really all over the place. I kinda gave up at that point trying to get a good sense of upper-limit.

[hun]

I'll start compiling my results into a table as well.  What do you think should go in a table? 

I record for every event of credit gain an index, a topic, a current relevant skill level, and resulting credits. But I am just playing my fortress for fun (rather than for pure testing purposes), so I don't think there will be a lot of data.  

I don't think it is a useful number since it is the same across skill levels and my initial calculation of 58.5 times on average seems to be accurate enough.

Yeah, I agree. It is too tedious to check it and your estimate seems good, so I do not track it.

[Scruiser]

Yes, this is because I am an idiot; the first entry from a list of topics doesn't show, and I haven't yet figured out why, but it is proly something stupid if my past programming experience has taught me anything.

My best guess is that it is some type of off-by-one error in the for loops you are using to access the knowledge names.  Do lua arrays/lists start indexing at 1 or at 0?  I think this might be the source of the problem... (I always get this confused when switching between matlab and python).  Judging by the for loops you used, I think you indirectly assume/imply that the indexing start at element 0, but according to an online tutorial I found, indexing actually starts at 1... (I found the tutorial here: https://www.tutorialspoint.com/lua/lua_arrays.htm)
So instead of:

Spoiler (click to show/hide)

Code: [Select]

for no, knowledge_type in pairs(knowledges) do
            entry = 0
            for name, known in pairs(knowledge_type) do
                entry = entry+1
                if (known==true) then
                    print("- ".. name)
                end
                if( knowledge_goal_category==no) then
                    for entry_g, known_g in pairs(knowledge_goal) do
                        if (known_g == true and entry_g==entry-1) then
                            print("- "..name.." (Learning: "..research_points.." / 100000 or "..research_percentage.."%)")
                        end
                    end
                end
            end
        end


Perhaps it should be:

Spoiler (click to show/hide)

Code: [Select]

for no, knowledge_type in pairs(knowledges) do
            entry = 0
            for name, known in pairs(knowledge_type) do
                entry = entry+1
                if (known==true) then
                    print("- ".. name)
                end
                if( knowledge_goal_category==no) then
                    for entry_g, known_g in pairs(knowledge_goal) do
                        if (known_g == true and entry_g==entry) then
                            print("- "..name.." (Learning: "..research_points.." / 100000 or "..research_percentage.."%)")
                        end
                    end
                end
            end
        end


I'll test this change out on my shadow clock researcher... and let you know if it worked...

a legendary critical thinker could get 25K research credit one cycle and 3K the next, so it is really all over the place. I kinda gave up at that point trying to get a good sense of upper-limit.

If enough people get enough data, we could make a histogram of the research credit gains and then see if their is a pattern.  My first guess would be a uniform distribution with the maximum limit set by skill level.  This would account for getting low credit gains even with highly skilled dwarfs.

I record for every event of credit gain an index, a topic, a current relevant skill level, and resulting credits. But I am just playing my fortress for fun (rather than for pure testing purposes), so I don't think there will be a lot of data.  

Sounds like a plan. I'll start a table myself.  If we can get a few other people to periodically collect data, we can at least get estimates of average good enough to put some usable advice on the wiki.

Edit: Nope, didn't fix the problem and in fact made the names come out wrong for other dwarfs.  Still, I think I understand the script well enough now to try a few other things to get it working...

Edit 2:  Got it fixed!  It was a bit hacky... I couldn't figure out where exactly the off by one error was, so instead I added an extra check for the case of the first entry in the topic list.
Here are the lines I added:

Spoiler (click to show/hide)

Code: [Select]

if (entry==1 and knowledge_goal_category==no and knowledge_goal[0]==true) then
print("- "..name.." (Learning: "..research_points.." / 100000 or "..research_percentage.."%)")
end


And the overall script  (note I commented out the stuff with skill levels because I can get that more easily from dwarf therapist):

Spoiler (click to show/hide)

Code: [Select]

-- View knowledge for currently selected unit
--[====[

read-knowledge
==============
View knowledge for currently selected unit

]====]

    local i,unit_name

    local unit=dfhack.gui.getSelectedUnit()
        if unit==nil then
            print ("No unit under cursor!  Aborting.")
        return
    end

    unit_name=dfhack.TranslateName(dfhack.units.getVisibleName(unit))

    print("")
    print("Printing data for "..unit_name)
    print("======================================")
    print("")
   
    histfigOffset = 0
   
   
    print(unit.hist_figure_id)
    histfig = df.global.world.history.figures[(unit.hist_figure_id+1)]
   
    for _, hf in pairs(df.global.world.history.figures) do
        if hf.name.first_name == unit.name.first_name and hf.name.language == unit.name.language then
            same_lastname = false
            for w, word in pairs(hf.name.words) do
                if (word > 0) then
                    same_lastname = (word == unit.name.words[w])
                end
            end
           
            if same_lastname == true then
                print(dfhack.TranslateName(hf.name))
                if histfig.id ~= hf.id then
                    print("Mismatching Histfig ID")
                    print("Unit: "..histfig.id.." Actual ID: "..hf.id)
                    histfigOffset = hf.id-histfig.id
                    histfig = hf
                    break
                end
            end
        end
    end

    print(dfhack.TranslateName(histfig.name))
   --[[
    for _, skill in pairs(unit.status.current_soul.skills) do
        print(df.job_skill[skill.id])
        print("\t"..skill.rating)
        print("\t"..skill.experience)
    end
   --]]
    if (histfig.info.secret ~= nil) then
        print("Books Read:")
        print("-----------")
        for _,book_id in pairs(histfig.info.secret.read_books) do
            book = df.global.world.written_contents.all[book_id]
                if (book.id==book_id) then
                    bookName = book.title
                    print("")
                    print("-\t"..bookName)
                    style = ""
                    for x, s in pairs(book.styles) do
                        sep = " "
                        if (x>0) then
                            sep = ", "
                        end
                        style = (style..sep..df.written_content_style[s])
                    end
                    print("\ta"..style.." ".. df.written_content_type[book.type])
                    if book.author > -1 then
                        maker = df.global.world.history.figures[(book.author+histfigOffset+1)]
                        makerName = dfhack.TranslateName(maker.name)
                        print("\tby ".. makerName)
                    end
                   
                    for x, ref in pairs(book.refs) do
                       
                        if (ref._type==df.general_ref_value_levelst) then
                            print(book.ref_aux[x] , df.value_type[ref.value], ref.level)
                        else
                            printall(ref)
                            print(book.ref_aux[x] )
                        end
                    end
                end
        end
       
        print("")
        print("Knowledge:")
        print("----------")
        print("")
        knowledge = histfig.info.secret.knowledge
       
        poetic_forms = histfig.info.secret.unk_v4201_1
        musical_forms = histfig.info.secret.unk_v4201_2
        dance_forms = histfig.info.secret.unk_v4201_3
       
        if unit.status.current_soul.performance_skills ~= nil then
       
            if poetic_forms ~= nil then
               
                print("Poetic forms:")
                print("-------------")
                for _, form in pairs(unit.status.current_soul.performance_skills.poetic_forms) do
                    print(dfhack.TranslateName(df.global.world.poetic_forms.all[form.id].name, true, true).."\t"..form.rating.."\t"..form.experience)
                   
                end
            end
           
            if musical_forms ~= nil then
               
                print("Musical forms:")
                print("--------------")
                for _, form in pairs(unit.status.current_soul.performance_skills.musical_forms) do
                    print(dfhack.TranslateName(df.global.world.musical_forms.all[form.id].name, true, true).."\t"..form.rating.."\t"..form.experience)
                   
                end
            end
           
            if dance_forms ~= nil then
               
                print("Dance forms:")
                print("------------")
                for _, form in pairs(unit.status.current_soul.performance_skills.dance_forms) do
                    print(dfhack.TranslateName(df.global.world.dance_forms.all[form.id].name, true, true).."\t"..form.rating.."\t"..form.experience)
                   
                end
            end
       
        end
       
       
        if (knowledge==nil) then return end
       
        knowledge_goal_category = knowledge.knowledge_goal_category
        knowledge_goal = knowledge.knowledge_goal
        research_points = knowledge.research_points
        research_percentage = (research_points/100000) *100
       
        print("Times pondered:", knowledge.times_pondered)
        print("")
       
       
        knowledges = {}
        knowledgeNames = {}
        knowledges[0] = knowledge.philosophy
        knowledges[1] = knowledge.philosophy2
        knowledges[2] = knowledge.math
        knowledges[3] = knowledge.math2
        knowledges[4] = knowledge.history
        knowledges[5] = knowledge.astronomy
        knowledges[6] = knowledge.naturalist
        knowledges[7] = knowledge.chemistry
        knowledges[8] = knowledge.geography
        knowledges[9] = knowledge.medicine
        knowledges[10] = knowledge.medicine2
        knowledges[11] = knowledge.medicine3
        knowledges[12] = knowledge.engineering
        knowledges[13] = knowledge.engineering2
       
       
        for no, knowledge_type in pairs(knowledges) do
            entry = 0
            for name, known in pairs(knowledge_type) do
                entry = entry+1
                if (known==true) then
                    print("- ".. name)
                end
if (entry==1 and knowledge_goal_category==no and knowledge_goal[0]==true) then
print("- "..name.." (Learning: "..research_points.." / 100000 or "..research_percentage.."%)")
end
                if( knowledge_goal_category==no) then
                    for entry_g, known_g in pairs(knowledge_goal) do
                        if (known_g == true and entry_g==entry-1) then
                            print("- "..name.." (Learning: "..research_points.." / 100000 or "..research_percentage.."%)")
                        end
                    end
                end

            end
        end
   
    end
   
    print("")


[PatrikLundell]

Lua lists created by the tools for list creation start at 1. However, Lua is capable of creating "lists" starting at any index, while allows it to map DF structures (which start at 0, as they're actually C ones).
Thus, if you want to create a list in Lua that maps to the indices DF use you'll have to do it manually.

[hun]

Sounds like a plan. I'll start a table myself.  If we can get a few other people to periodically collect data, we can at least get estimates of average good enough to put some usable advice on the wiki.

Yeah, although it is going to be slow on my side for sure, and I don't even know if I collect enough data worthing any analysis. There is a lot of randomness, as far as I can see. But probably still less than a uniform distribution.

Edit 2:  Got it fixed! 

Thanks, now it is OK with most of the dwarves. But I still have one guy for whom I cannot see the progress.

[Scruiser]

What is the topic they are pondering that you can't see progress for?  Worst case scenario, I think I could make the script check for that exact topic in an additional if statement.  Not a very elegant way of coding, but it works...

[hun]

The problematic topic is "Reproductive Behaviour". Although I have no idea whether the problem is in the topic or in the dwarf that studies it.

[Scruiser]

I ran into an error myself.  I had been using the scripts on quick test forts... when I tried it on an existing for I had that has lasted much longer with a much bigger population, I got the following error:

Code: [Select]

[DFHack]# read-knowledge

Printing data for Asen Enolkadol
======================================

101047
...8\Dwarf Fortress 0.44.12/hack/scripts/read-knowledge.lua:29: Cannot read field vector<historical_figure*>.101048: index out of bounds.
stack traceback:
        [C]: in metamethod '__index'
        ...8\Dwarf Fortress 0.44.12/hack/scripts/read-knowledge.lua:29: in local 'script_code'
        ...k 0.44.12-r08\Dwarf Fortress 0.44.12\hack\lua\dfhack.lua:680: in function 'dfhack.run_script_with_env'
        (...tail calls...)

The number after vector<historical_figure*> changes depending on which dwarf it is, but all the rest of the error message stays the same.  I've tried it on a starting dwarf, migrants, visitors, and historical figures and I get the same error message.  I imagine the index out of bounds indicates something is too big... anyone have any ideas?  Whats the largest/oldest fort you've got this script to run on successfully?  This fort I am trying it on is a little over 7 years old, in a world that is 1463 years old, and the fort has a population of 138.