Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[thistleknot]

Maklak's concern over skill % jumps can be accommodated.

On an 8 year fort, I had 2400 skills at ~60 skills wide (when I was initially doing my numbes); in actual gameplay, there's ~119 skills wide.

So with ~50 dwarf's, I had a big matrix, ~2400

I only had 128 Unique Skill Values.

The mean was 50%, the median was 46% (in my tests)

~<95% of the values were below 50%.

That was because the median was 46%, and was padded from left to right by 45% on either side.

Approximately 90% of the values around the median were taken up around the median's distribution within the total number of 2400 items.

So... that means no other items would be lost if you were to move values below a .5 threshold if 90% or more of the values were below .5

Simply finding that gap, say by testing the count of values below .5, if it >=90% of the distribution, then cut it out;

leaving only the values above .5 mean.  I don't know if it's that important, but considering the extremeness of skewed data and transformation.   threshold should remove low values from a dataset.


**Another idea is to not remove the low values, but somehow stretch down values?  I personally don't think it's that big a deal, it's better than under-representing skills, but if those above .5 values were stretched down to .5, and the lower than .5 were reduced by an amount that would achieve an overall .5 mean.**

Personally I haven't experienced too much headache with skills being an issue; however I have had some issues with the attribute potential weight set really low at .05, and the skill rate weight set to .05... I get skills that are 5-16 levels higher than another getting passed up (at low levels).  Only happens when I set those two weights, and global skill weight far outweighs any other weight.

[Rallor]

So which version of Qt am I supposed to compile it with on Windows? When I use 5.3 with mingw and try to run it I get segmentation faults.
Edit: I got it to work once I coped the etc folder from the source to the folder of the executable. This instruction was on the google code site which is no longer maintained and is not on the github build instruction. I recommend adding it.

[thistleknot]

I don't know if you're still in on the conversation or not Maklak.

But we identified something with your original skill simulation formula Splinterz and I were discussing.

Part of the formula works with the assumption that we should be capping things at level 20.

However, other parts of the formula are not capped.  Resulting in simulated levels being returned of 30 vs a raw level of 1

which is causing havoc [well in some examples] when using your formula.  As a level 8 and level 18 are passed up by lower %'s when doing a labor optimization.  I've verified it's with the skills specifically and the skill rate weight, otherwise my weights work appropriately.

Splinterz pointed out that the simulated formula returns values above 20 (as high in 30 in my case).

So, I thought since your initial desire was to stop comparing once legendary is achieved, shouldn't the values reported by the function also only consider a max of level 20 in all instances as well?

Here's reference code and the variables I was thinking about

Spoiler (click to show/hide)

are we basing sim_xp on 29000 or the highest xp a player has?  If it's the highest xp a player has, then it's reporting a value not capped at 20.

Also.

does get_double_level_from_xp cap at 20?  If so... then one part of the formula isn't capped at 20 and the other part is.

In fact, if any number in this formula that returns a value that is later based on a level, should be capped at 20.

Code: [Select]


double simulate_skill_gain(int xp, int rate)
{ if (xp >= max_xp)
    return 20.0 / 20.0;

  if (rate == 0)
    return get_double_level_from_xp(xp) / 20.0;       // Obviously stays the same.

  int sim_xp = max_xp;           // 29k seems like a good value to me.
  sim_xp = (sim_xp / 100.0) * rate;    // This is how much XP will go towards skill learning.
  int total_xp = sim_xp;
  double ret = 0.0;
  //int curr_level = get_level_from_xp(xp);
  double curr_level = get_double_level_from_xp(xp);
  int curr_xp = xp;

  while ((sim_xp > 0) && (curr_level < 20.0))
  { int xp_gap = xp_levels[int(curr_level)+1] - curr_xp;         // How much XP till mext level?
    if (xp_gap > sim_xp)
      xp_gap = sim_xp;
    ret += xp_gap * curr_level;
    curr_level++;
    curr_xp = xp_levels[int(curr_level)];
    sim_xp -= xp_gap;
  }

  if (sim_xp > 0)
    ret += 20 * sim_xp;
  ret /= total_xp;
  ret /= 20.0;
  return ret;
}

However, I am also working on a transformation formula for skills in general.  Above Median = 50% to 100%, and <= median equals the old value factored by a number that targets an overall mean of .5, this should address the issue with [>median] skills all being grouped at 95% to 100% range.

Update:
I got a formula that transforms the lower range <=median (post ecdf/rank %).

These numbers work with the ecdf/rank% we derive, not the raws.

x = x,y in our skills*dwarfs matrix

It's based on the : sumif(x > median) / total count (aka dwarfs * skills).
difference from the same formula post conversion for >Median (which we derive first).

So this difference measures the % change in the upper (aka >Median) values, that needs to be carried over when transforming the lower (aka <= median) values, so we double the difference [of the means], let's call this factorValue.

factorValue = 1 /factorValue for values <=median.

This preserves ~.5 mean and spreads all the >median skills to 50 to 100%, and those values that were <= median are 0 to 50% respectively.

I sent Splinterz a spreadsheet breakdown (sorry, I don't have matlab, closest thing I could get my hands on would be r-project, but I don't think it's necessary).

I think we could auto check each category,

by checking if min = median,

 then we run this double transform.  Hell, variations could be used to transform in other methods if needed, such as if min = 1st quartile, we could derive a way to fit each quartile range into it's own distribution transform.  Luckily, so far the mixture models that have shown themselves only have 1 extra spike in the data, ~0 for skills and preferences, and the rest of values (aka >~0).  So we're kind of splitting the ecdf/rank % into two separate distributions that we throw into a 0 to 100% range.

An opposite transform could be used for negative skews if we ever encounter any.

In the end, this allows us to "flatten" a distribution from 50 to 100%, aka, the %'s are evenly spread, just as they are with all attributes with this given range, and we're still centered around the median.

Update:

Here's some psuedo code on the method

Spoiler (click to show/hide)

Code: [Select]

//so skills are converted into %'s using rank% averaged with ecdf%, this becomes our SkillsECDF[] <list>.

//variables
list SkillsECDF[];
list overMedianSkillsECDF[];
list underMedianSkillsECDF[];

int totalSize = SkillsECDF[].sizeof();

float averageOfOverMedianOfTotal;
float averageOfUnderMedianOfTotal;

float lowerListFactor = 0.0

bool runFlag = 0;

//test to see if we even need to perform this deskew?

//skillsECDF-Median = SkillsECDF[].median();

if ( ( SkillsECDF[].median() == SkillsECDF[].1stquartile() ) && ( SkillsECDF[].min() != SkillsECDF[].max() ) )
{
runFlag = 1;
}

if (runFlag)

//split distributions up in two (overMedianSkillsECDF & underMedianSkillsECDF)
{

//sort list based on % value (important for last function)

//name references dwarf's name)
SkillsECDF[].sort(name)

//var: SkillsECDF[].median()

float median = SkillsECDF[].median();

//update overMedianSkillsECDF with upper values of SkillsECDF

//run through SkillsECDF
for (int x = 0; x < SkillsECDF[].sizeof(); x++)

{
//keeps track of size of overMedianSkills[]
int y =0;

//keeps track of size of underMedianSkills[]
int z =0;

//upper, overMedianSkillsECDF
if ( SkillsECDF[x] > median )
{
overMedianSkillsECDF[y] = SkillsECDF[x];
y++;
}

//lower, underMedianSkillsECDF
if ( skillsECDF[x] <= median)
{
underMedianSkillsECDF[z] = SkillsECDF[x];
z++;
}

}

//derive means of each new distribution against total count of SkillsECDF[]

averageOfOverMedianOfTotal = overMedianSkillsECDF[].sum() / totalSize;
averageOfUnderMedianOfTotal = underMedianSkillsECDF[].sum() / totalSize;

{

//local var's

float upperListNewMeanofTotalOld = overMedianSkillsECDF[].sum() / totalSize;
float upperListNewMeanofTotalNew;

//transform upper list
for (int x = 0; x < overMedianSkillsECDF[].sizeof(); x++)
{
//countif (list, criteria), works same as excel function
//rank (list, value, order), works same as excel function

// divides by 2 to scale to 0 to 50%.  Add .5 to scale back up to 50% to 100%.
//basically a rank% + ecdf %  averaged , divided by 2, and .5 is added.
overMedianSkillsECDF[x] =

(
( rank(overMedianSkillsECDF[], overMedianSkillsECDF[x], 1) / overMedianSkillsECDF[].sizeof() )

+

(  countif(overMedianSkillsECDF[], "<=" &overMedianSkillsECDF[x]) /  (overMedianSkillsECDF[].sizeof() ) )

/ 2
) + .5;

upperListNewMeanofTotal = overMedianSkillsECDF[].sum() / totalSize;

lowerListFactor = 1 + ( (upperListNewMeanofTotalOld - upperListNewMeanofTotalNew) * 2);

}

//transform lower list
for (int x = 0; x < underMedianSkillsECDF[].sizeof(); x++
{
underMedianSkillsECDF[x] = underMedianSkillsECDF[x] * lowerListFactor;
}

//replace values of SkillsECDF[] with the values generated by underMedianSkillsECDF[] & overMedianSkillsECDF[] respectively

{

/*

we then replace the values in SkillsECDF[]

with the values we generated in: underMedianSkillsECDF[] & overMedianSkillsECDF[];

so, they'll have to be mapped to a name pre/post conversion.

*/
//remember the list was sorted prior to conversion, so this should map back based on name

for (int x = 0; x < SkillsECDF[].sizeof(); x++)
{

//loop thru the upper list
for (int y = 0; y < overMedianSkillsECDF[].sizeof();
{
if (SkillsECDF[x].name == overMedianSkillsECDF[y].name)
{
SkillsECDF[x] = overMedianSkillsECDF[y];
}
}

//loop through the lower list
for (int y = 0; y < underMedianSkillsECDF[].sizeof();
{
if (SkillsECDF[x].name == underMedianSkillsECDF[y].name)
{
SkillsECDF[x] = underMedianSkillsECDF[y];
}
}

}

}



}





}


Here's a pic of it in action

http://imgur.com/rls1joX

and a link to my dffd sheet of it in action

http://dffd.wimbli.com/file.php?id=8705

Btw, I double tested this with two different data sets, one very large ~2400, and this one with ~35 values.  Both times my targeted mean of within .005% from 50%.

[Maklak]

Since I got 2 PMs, I just dropped here for a moment and skimmed the last post.

DF wiki once said something about skill levels having a cap at legendary+5 and that quality doesn't improove beyond lvl 20 skill. So I assumed it is reasonable to cap skills there, even if the XP says "level 35" or something else ludicrous.

> However, other parts of the formula are not capped.  Resulting in simulated levels being returned of 30 vs a raw level of 1
I'm not sure what you mean here?
* If function simulate_skill_gain returns more than 20.0, then there is an error in code somewhere in there and debugger should help find it.
* Attributes? Hard capped at 5000.
* Traits? Capped, either yes or no.
* Personality? Capped, 1-100.

> As a level 8 and level 18 are passed up by lower %'s when doing a labor optimization.
For the same learning rate, the function should be monothonically increasing with XP untill lvl 20, where learning rate no longer matters. Maybe theyse guys are passed because they have weak attributes, or too many labours already enabled?

> Splinterz pointed out that the simulated formula returns values above 20 (as high in 30 in my case).
This shouldn't be so. Can you give me the input data for which it fails? And value of max_xp? Besides, it divides by 20.0, so it should return values in <0, 1>

By the looks of it, the easiest fix would be to replace the last line "return ret;" with "return ret <= 1.0 ? ret : 1.0;" but that wouldn't really fix a bug, just mask it.

As for transformation in general, what do you want to accomplish? What skill level at 100% learning rate should return 0.5? Once you decide, it can be adjusted with choosing the right xp_sim, but that's a bit more involved than a quickie I'm writing here. It needs bisection on a weighted mean of levels where their weight is how much XP you need to get to the next level. More or less.

> are we basing sim_xp on 29000 or the highest xp a player has?
sim_xp should be a constant. Sensble values are in <5000, 29000> range.

[thistleknot]

thanks Maklak, not sure what Splinterz did, but he found the culprit and fixed it (although I'm curious what he did exactly).

Skills/Preferences all now scale from >Median to 100% to 50% to 100% respectively.

Anything that was >=median is now below 50%.  It allowed for us to deal with skews.

The >=Median values are still based on an ecdf/rank% (so are the upper values), but the lower values are factored [up] by a derived mean difference from the new >median values, and the the old >median values against the total Count (aka the whole dataset, not just the segmented one).

Complicated stuff, but it worked, and moved our data back to .5 mean (which needed to happen after I cut that gap out), it effectively cut the gap in %'s from the large # of ~0 [skewed] skill/preference values, which was ~46%, and the high %'s listed at the other end. (~95%+).

So now the lower distribution looks approximately the same, but now we cut that gap out.  So now the upper distribution maps down to 50%+.

The best part is, non 0 (the check actually checks for >Median values, but only runs if 1st Quartile = 3rd Quartile) skills are counted as 50%+ to 100% based on ordinal ranking.

It was my second transform and I think it worked out beautifully.

Here's how it's looking right now.

http://imgur.com/ZiLxamy

[thistleknot]

Spoiler (click to show/hide)

So beta testing current results and it seems to be working as expected.

I noticed one minor caveat though.

Due to the way we measure values based on their ordinal position.  The next value beyond itself is only measured by it's ordinal position to the value just below it.

I figured with a large enough matrix of values, we can iron out any anomalies.

Problem is... skills aren't really "random".  Although, I figured this wouldn't, shouldn't matter, it's behaviour as being skewed and based on guided values (i.e. a player cultivates his dwarf's skills, usually from a 0 value; sometimes outsiders come in with their own non zero values).

Point is.  There are a lot at 0 value, and a lot at lower level values.

There are a few at higher levels.

I had a 2 year old fortress.  I had legendary dwarf's that had matching high % ratings.

but for some reason, I had a level 2 (fishing) dwarf that was recorded as 90% for his skill rating.  I had a level 13 ~98%.

So I set the weight of skills to 1.5 and disabled everything else (effectively 100% weight), and I found that they were all mostly level 0 and level 1's, which ranged from 50% to 70%.

Since the majority of dwarf's (aka large proportion of my distribution) were these 1 level dwarfer's, they took the majority of the %'s in our ecdf-rank calculation.  So level 2 was actually quite rare, and was reported as 90%, but this is where the problem occurred. 

When I ran a labor optimization, I had a level 13 dwarf passed up for lower level dwarf's, It didn't make sense, but when I looked at the output in the log file, I realized this gap in %'s was due to the lack of upper ranging values. 

I'm looking at the breakdown of the #'s.  I had an idea, but it's just an idea, and it kind of goes against the whole rank-ecdf version I came up with.  However, it might be possible, to measure the difference inbetween values and store it by factoring it against the rank/ecdf position.  Each value would accumulate to a range from min to max that should be proportionate to the true range between min and max.

Instead of normalizing on the count size of the matrix, we can normalize (aka divide by) against this new derived range that we created.  It's almost like a minmax version of ranks * distance between raw values.

It's a bit much (you have to break the grid down into a sorted list, then derive not just the rank/ecdf position of a value, but it's difference from the last lower value), then you'd factor these two values together to arrive at a cumulative value that you would keep adding (we would actually derive a cumulative % value).  Once you reach max, that becomes our ~100% marker.  Min will be ~0%.  This would solve the problem of low values taking up a large % of the distribution from taking up even more % (of which we already tried to resolve).  The process could be sped up to, as 0 values (aka values that have no difference inbetween themselves at the lowest point in our distribution), would have a difference of 0, so it's rank * 0 would be 0%!  so we could skip a lot of skills by checking if value == min, skip,

when value != min, then derive the distance from the prior value, and calculate it's rank/ecdf position * difference = cumulative value

I think it might work to retain the difference in original values.

It also would be hard to measure the difference between original values of preferences, as they are binary.

So far the idea is to derive a fraction (aka %) of the number of preferences flagged / number of preferences in a given role.  Then these results are all compared to each other per dwarf, per role in it's own matrix.  The only difference in values I have for that, is the distance between these fractional values, but that should work, and it would do some amazing things I think in retaining original value, and ties in values would have 0 difference between values, so effectively it would remain at the same cumulative %, until we reach ~100%

derp, I just realized that's recreating a minmax distribution, and will not have a .5 mean...

damn.

Update:
I have an idea, and it involves using a rank/ecdf% averaged with the minmax% just for >Median values of skewed data.  I can still target a .5 mean, and address this level 3 vs level 13 guy issue I'm having.  I can even test if this conversion is necessary by checking the >Median values, and seeing if that specific dataset's 4th quartile is > 3rd quartile by over 10 times.

[thistleknot]

http://imgur.com/0GNl0eI

Imagine above picture defines our issue with skills and their skew.

We used ordinal rankings (orange) line pretty much for values >Median transformed to a 50% to 100% range.

Problem was, a lot of low level skills take up the majority of the distribution (light blue line shows how skills really look, aka min/max conversion).

Since we pretty much put skills in the above 50% range for values that are >median (aka non 0 values).

We can average the rank-ecdf% with the minmax% and arrive at a much smoother better line (dark blue) that works much better than the rank-ecdf% line.

Now the problem with my level 13 being passed up by a level 2 dwarf has been fixed.  There are still some minor discrepencies, but now it's much easier to correct with weights.

The only sad thing, was it was really hard to achieve a .5 mean, but we got close, a .48, lowest was a .46 mean for the distribution.

We derive a [targetted] mean by transforming values <=Median, which in skills is ~100% of the time the value 0 (unless you take into consideration the skill rate weight transformation, used for mods).  But this method we use to check for behaviours of skewed distributions, we can treat even these values fairly, that are <=median.  As the median value still is ~50%.  Best part is, we preserve relevant information of values <=Median even if 0 is not median.

I proposed we set a weight, so a user can value a minmax vs a rank-ecdf%, very similar to how we have weights for attribute potential, as well as skill rate weight, but... it's a little overkill.  As long as we achieve a natural progression for lowest skill to highest skill.

we had to derive some tests though.

Derive quartiles of the values >Median, take their raw values.  If (max/3rq Quartile) > 5 (or 10, not sure what Splinterz set it to), then we do this transform on skills.  It get's passed appropriately for our preferences, since they don't have this exponential curve (aka light blue line).

So... I think it's almost ready.

The big test was if this last method would work, and so far it's looking great.

There's some minor tweaks I proposed for the flags that are used to run these transformations.

One flag is if 1st quartile = median, then we do the transform that pull down to 50%.  I was proposing to modify the test if 1st Quartile = Median (position + 1).

We have a followup flag.  If the original data's >Median values Max/3rdQuartile > 5, then we do the minmax average with the rank/ecdf%.  I was proposing to modify this by if Max (position - 1) / 3rd Quartile > 5, then we do the minmax average with the rank/ecdf%.

But... I think they might be good enough.

Complicated stuff.

Update
Btw,  I think this latest beta/alpha (v13 I think), is working perfectly.

[ExpHP]

A couple of remarks:

The good

• Builds relatively painlessly on Ubuntu 12.04 LTS (Precise).  Qt5 is not available on the default Precise repositories, but users can obtain access to all the required packages by adding the following repository (don't forget to apt-get update afterwards):

Code: [Select]

sudo apt-add-repository ppa:ubuntu-sdk-team/ppa

• Looks beautiful, works great.  It's quite an improvement over the old version.

And the bad:

• Having it install to /usr/bin is a bit... uncomfortable for me, but thankfully all of the files added have "dwarftherapist" or "DwarfTherapist" somewhere in the name, so collisions with existing files are unlikely. (on ubuntu, /usr is typically reserved for packages from the repositories, and stuff built from source is installed to /usr/local.  I'm not sure whether or not qmake offers any easy way to allow the user to set an install prefix, but it looks like the paths in the .pro file are hardcoded anyways)
• Having it write a log file to /usr/bin/log/run.log is quite uncomfortable.  (namely because of the possible existence of /usr/bin/log as an executable)
• /usr/bin/dwarftherapist (the thing you're supposed to run) is not marked executable, and thus cannot be run directly from the command line.  Easy to fix, but mildly confusing (since /usr/bin/DwarfTherapist is marked executable).

[thistleknot]

really excited about something that I spent a lot of time on yesterday.

I think I was able to approximate a wave of the [values of the] original distribution, then mean adjust it to .5 from a scale from 0 to 100%

Check it out

Attributes:

http://imgur.com/2Vyaa5a

So far, it should work for attributes, traits.

What this method does is accounts for difference between values and not just rank value.

It's done by transforming the center of the distribution around the mean, then the median, then mean adjusted by additive value to .5.  Then transformed again from 0 to 50 to 100%.

Doesn't work for skills though, and that's due to their explicitly skewed, exponential behavior; but we can take advantage of the rest of the comparativeness between values with this new method, and apply it to our force.

The median value tends to hug near the mean, so I prefer this method over the flat distribution curve of the rank-ecdf method.  I didn't notice an issue with the [rank-ecdf] method until my fort got about 5 years old, and I noticed my warrior's attributes had trained up really high and it wasn't really accounted for.

Update:

splinterz in his amazing machine way, coded up the changes again, addressed an issue with skills, and we almost completely removed the presence of rank-ecdf an opted for more natural curves, as seen with the picture shown above.  It's more akin to a minmax method transform; but we do multiple transforms.  First around the mean from 0 to 50% to 100%.  Then median from 0% to 50% to 100%, and then we mean adjust the distribution to .5, then we shrink/grow the ends to match 0 to 50 to 100%.  The affects on the distributions are minimal.  The only distributions that get this type of transform are attributes, traits, and preferences.  Skills get a regular minmax /2 + .5 to get a 50% to 100% range for >median values.

Here's a pic of roles drawn

http://imgur.com/j8T8eTQ

[tryrar]

So, new version released today. Does Dwarf Therapist work?

[thistleknot]

lol, not yet.  Memory location updates would have to take place now.

[Vabalokis]

DF2014 is a good reminder that playing df without therapist is pain in the ass

[Waladil]

I know 2014 just hit, but how long is it expected for the new memory maps to be figured out? Is that like a week-long deal, or is it "yeah, we can pound those out in about half an hour!" kind of a situation?

[kleril]

Dunno how it is for larger updates, but 34.x to 34.x+1 took a day, maybe two. Fingers crossed for tomorrow.

[splinterz]

and so it begins...