Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Silverionmox]

People trying to push their favourites isn't bad per se, if the goal of the list is to decide which items to work on first. If the goal of the list is to decide which items to include or not to include in the game eventually, it might be harmful. However, people picking favourites will be against their own interests if they actually want other stuff included.

[Toady One]

Yeah, I think the two voting systems will complement each other well in that way.  I expect fractional voting will end up looking more-or-less like the current list (since 3 votes restricts you to a few options), whereas a, say, 1-10 rating system would give us a completely different list based more on the overall sense of what people want the game to be, which will be interesting to see.

I'm not sure what direct compare gets you...  maybe an average of those two visions, sort of.  If direct compare uses an ordered list for each voter, then it's like a fractional vote where everybody has to smear their amount out in the same proportions (but on different items of their choosing).  This reduces the "what I want most" feeling and injects some of the overall game feeling into the final results.  There's a tradeoff between forcing people to pick one thing over another and letting them tell you how much they like something in and of itself without having to devalue anything, but I'm not sure how to understand the final result for direct compare in terms like "what people want now" or "the overall finished game".

[corvvs]

Yeah, I think the two voting systems will complement each other well in that way.  I expect fractional voting will end up looking more-or-less like the current list (since 3 votes restricts you to a few options), whereas a, say, 1-10 rating system would give us a completely different list based more on the overall sense of what people want the game to be, which will be interesting to see.

I'm not sure what direct compare gets you...  maybe an average of those two visions, sort of.  If direct compare uses an ordered list for each voter, then it's like a fractional vote where everybody has to smear their amount out in the same proportions (but on different items of their choosing).  This reduces the "what I want most" feeling and injects some of the overall game feeling into the final results.  There's a tradeoff between forcing people to pick one thing over another and letting them tell you how much they like something in and of itself without having to devalue anything, but I'm not sure how to understand the final result for direct compare in terms like "what people want now" or "the overall finished game".

I think direct compare is the most useful in terms of getting at the heart of what is the most interesting overall but also the most complicated to process.

A two step system would be the way to do it, I think.

First loop, for each individual person sort voting items by preference. Now turn those into point scores (signed - the median item voted on is set to zero points. items lower are assigned a negative number based on distance, higher items are given a positive number. The total range should be dependent on thenumber of total voting items, regardless of the number of votes cast by each individual; that way the person who only cast 3 votes still gets his highest rated item at +127 while the person who compared all 255 items at some point in the voting also has their highest item at +127).
Second loop, tally preferences for each item by adding everyone's individual scores. Higher scores are what have been consistently voted as more important than other features.

Complications can arise when one person votes such that item A is preferred over item B which is better than item C which is better than item A. This can be fixed by rigging the questions so that items are never compared to items of sufficiently different rank so as not to allow those answers. Once an item has been voted on for the first time by a given voter, only allow him to compare it to items immediately surrounding it in rank, or items not yet voted on by him.

Another solution would be to allow the totally random questions and force all items involved in a loop like that to zero points, but I think it's better to force the questions to extract meaningful answers.

[Granite26]

Don't we have enough voters that gaming the system starts to go away?  I mean, to a certain extent, voting for marginal 'do next' items will affect the order, but shouldn't I be ABLE to say 'I want X more than Y, but Y and Z are the two popular choices so I'll throw in for Y first', because that's the best way to spend my vote-currency?  The 2-1-1-1-1 example is correct, but when the sums get into the hundreds, the gaming just hurts the player's secondary choices without making his primary one look better.

I'm leaning towards an X votes (3-10) for different things (so you can't load up)  IF you do want to load up, I still think arbitrary cuts is more mathy than useful... Just use 100 votes and leave it at that.  (Programmer says: make the number of votes a variable and multiple votes for one thing a checkbox)

combined with a Up/Down/Apathy (with shown counts, so you can tell controversial stuff from general apathy).  Scores should be -X, 0, and +Y.  X and Y don't need to be the same, but X(dislike) should be less than the 'no opinion' or 'hasn't voted' options, and since 'hasn't voted' should be 0, X needs to be less than zero for the summation.

And instructions for how vote-advice will be interpretted  (Up/Down is vision, quantified votes is timeline) would be useful, so people knew how it was going to be looked at.  Especially if its a 'timeline will be interpreted for work required'.


Edit:  Edit for clarity

[zagibu]

I tried to understand what all of you said, but it was too complicated for me.

I think that the fixed votes methods (pick1, pick3, pick10, whatever) result in a more or less exponential curve, were few top suggestions attract most of the votes and the rest has to be content with a lot less. This is because people are forced to choose a subset of their preferences, and then they choose those that are also popular among others, because these have the highest chances to top the list. What I'm not sure about is whether more votes increase or decrease the steepness of the curve (probably decrease).

The fractional method and the up/down or /up/neutral/down voting should produce a much flatter curve, because here, people can vote for any number of things. Of course, the fractional method could turn into a pickX farce, because people realize that the more things they spread their vote on, the less power each single item receives. The up/down method though is different, because each up is 1 vote, so it doesn't make sense to vote for only your favourite thing. It's more like a dual/triple partition of the list into things you like, things you don't care about (if neutral vote is included), and things you dislike. It might be pretty hard on some suggestions.

The direct compare method on the other hand is hard to predict. With our 200 suggestions till now it would lead to max. 20000 comparisons PER USER. with 1000 Users voting, this would lead to 20 millions entries in the db. This means it won't be possible to dynamically calculate the rankings on every page load. So it will definately be necessary to calculate a ranking already at the comparison level (like add 1 for every "win" in a pair).

I have the feeling that the direct compare method would be the most accurate in representing the demands of the community, because it kind of let's everyone construct their private ranking, then calculates the average of those lists. Also, it honorates hard work, because those who do more compares also have a stronger influence on the total outcome.

What I also thought of was a fractional ranking, where the total weight of a person's vote is determined by their forum post count...maybe in some limits, but maybe also direct (so that Granite26's vote would be equal to that of over 3000 newbies...wouldn't you like that, mate?).

[Granite26]

Yeah, but I talk too much... that's the long and short of that There's better ways to rank people, although most of them should be Toady-private.

As far as the direct compare goes, in this context it doesn't make sense to worry about circular results (Prefer A to B, B to C, and yet C to A), so direct is equivilent to an ordinal ranking system per player, with is O(NlogN) if you use a quicksort algorithm to generate questions (460 questions?), especially after you generate your 'generic rating'.  As far as storage, you'd only need to store 200 ordinals (uints) for each user.

This only works if you agree that it doesn't make sense to prefer improved sieges to improved hauling, improved hauling to improved graphics, but yet prefer improved graphics to improved sieges.

(A quicksort type structure is also nice because a bubblesort set of questions would be BOOOOORING) Edit - After reviewing quicksort this isn't actually true... still, some sorting algorithm could be found that asks slightly more questions with a better variety.

Edit 2: - Up/Down also allows you to see the difference between the sexy stuff people get excited about and the simple engine stuff that everybody just wants to show up.  Perhaps show 'average vote' as a variable in the multiple vote system?

[Draco18s]

This reminds me of some book I read where this family was trying to sort out all of the stuff that an elderly couple left behind.

One of the teens came up with this ingenious idea, lugged everything out to a parking lot and had each person come by at a different time and do the following:

Grab something and move it that way it you want it more because of its monetary value (X) and that way if you want it due to sentimental value (Y).  Move it in the other direction for not-wanting it or considering it of no value.  He then measured the locations of all the stuff from each person and used a computer program to figure out who should get what stuff.

One of the questions he was asked as, "what if someone puts everything in the far corner of the parking lot?"  "Then it's as if they made no choice at all: they want everything equally, so it doesn't matter what they get."

Based on this, ever suggestion could have two types of vote:
1) Good idea/Bad idea ("this should be in the game because it makes the game good," e.g. more flexible RAWs, more creatures, a use for soap)
2) I want this idea/I don't want this idea ("this is my pet, I want it because I think its cool," e.g. more realistic mining, poop, better wood use, harder/realistic farming)

And they could have a sliding scale 1-10.  Ideally, items rated highly on the #1 scale are core items, #2 scale things are bloats and powergoals (reqs are half-and-half).  The #2 scale is for how passionate a user is for wanting to see that feature, whereas the #1 scale should be more reflective of how it effects vanilla gameplay (being able to disable and dodge traps in adventure mode is something that's needed but might not be highly desired for some users because they primarily play in fort mode, and want to see more realistic mining or farming, whereas for someone else, they might think that realisic farming is not only desirable but necessary for a proper challenge, but also sees the need for adventure mode trap dodging, but is indifferent (a 5 on the 1-10 scale) about when the feature comes).

What the math has to do is find the average location on the XY plot based on the input (each user's input scaled appropriately to a bell curve--if someone rates everything a 8 with a single 2, then on average they've rated things as a 5, except one item as a 1) and give it a score, the higher the score is on both scales would indicate a higher priority on the dev list.

It separates out why people want different features and finds the features that the game needs while supplying features that people desire.

[zagibu]

Hmmm, I'm not sure what the use of the 2 dimensional scale is, if you compress it to 1 dimensional again.Also,  I guess everyone would vote their things highest on both scales, regardless whether it is a "needed" or a "liked" thing.

[Granite26]

Sic:  Lie

[Draco18s]

Hmmm, I'm not sure what the use of the 2 dimensional scale is, if you compress it to 1 dimensional again.

Nono, it stays as a two dimensional scale the whole time.  The end output has each suggestion having an X and a Y value and can be plotted on a Cartesian grid.

Also,  I guess everyone would vote their things highest on both scales, regardless whether it is a "needed" or a "liked" thing.

Voting everything high is the same as voting everything low: you have no preference.

Spoiler: "Example" (click to show/hide)

Using a 1 dimensional scale to show how it works

Urist:
Item 1: 10
Item 3: 10
Item 3: 10
Item 4: 10
Item 5: 10
Item 6: 10
Item 7: 10
Item 8: 10
Item 9: 10
Item 10: 10

Mean: 10, Median: 10, Mode: 10
Adjusted bell curve:

Item 1: 5
Item 2: 5
Item 3: 5
Item 4: 5
Item 5: 5
Item 6: 5
Item 7: 5
Item 8: 5
Item 9: 5
Item 10: 5

Cog:
Item 1: 1
Item 3: 1
Item 2: 1
Item 4: 1
Item 5: 4
Item 6: 4
Item 7: 4
Item 8: 5
Item 9: 9
Item 10: 10

Mean: 4, Median: 4, Mode: 1
Adjusted bell curve:
Item 1: 2
Item 3: 2
Item 2: 2
Item 4: 2
Item 5: 5
Item 6: 5
Item 7: 5
Item 8: 5
Item 9: 9
Item 10: 10

I'm not using exact math there, nor a very large sample.  But the idea is to look at each person's data set and find the standard deviations and adjust the data appropriately to get a normalized data set.  Someone who votes only 4/5/6 (down vote/neutral/up vote) would have their data internally represented as 1/5/10 when compared to other voters.
(I am not implying that a user's votes are not changed visibly to them, as every new vote would alter the internal representation.)

It's in a person's best interest to vote using the whole spectrum available, but the system compensates for users who don't, so that a 4/5/6 d/n/u is exactly the same as a 1/2/3 and an 8/9/10, as well as preventing an abuse of the system (i.e. voting all 10s).  It attempts to separate out why a user wants a particular feature, and while all users might not use the want/need axis the same way, there should be enough variation to give the output a range showing which features are highly desired and which are needed more quickly.

[zagibu]

If it was not me having to actually implement this idea, i'd be all for it. But this just scares me. Plotting 200 suggestions on a 2d plane? Also, the separation of needed and liked seems a bit artificial for me. A game should provide fun, and usually, gamers suggest things that they think could lead to more fun, be it UI improvements or new features. Most gamers probably aren't well enough versed in game design to vote things needful but disliked.

[zagibu]

I have decided to not reduce the number of votes in the pickX voting mode (the one that is currently up) but make X variable, so that it's up to Toady or the community to decide how many fixed votes are needed. It would be possible to start with one and increase it each month or so...
I've also completed the thumbs up/neutral/down rating mode. I'd say I'm 30% done overall.

This reminds me of some book I read where this family was trying to sort out all of the stuff that an elderly couple left behind.

One of the teens came up with this ingenious idea, lugged everything out to a parking lot and had each person come by at a different time and do the following:

Grab something and move it that way it you want it more because of its monetary value (X) and that way if you want it due to sentimental value (Y).  Move it in the other direction for not-wanting it or considering it of no value.  He then measured the locations of all the stuff from each person and used a computer program to figure out who should get what stuff.

One of the questions he was asked as, "what if someone puts everything in the far corner of the parking lot?"  "Then it's as if they made no choice at all: they want everything equally, so it doesn't matter what they get."

Based on this, ever suggestion could have two types of vote:
1) Good idea/Bad idea ("this should be in the game because it makes the game good," e.g. more flexible RAWs, more creatures, a use for soap)
2) I want this idea/I don't want this idea ("this is my pet, I want it because I think its cool," e.g. more realistic mining, poop, better wood use, harder/realistic farming)

And they could have a sliding scale 1-10.  Ideally, items rated highly on the #1 scale are core items, #2 scale things are bloats and powergoals (reqs are half-and-half).  The #2 scale is for how passionate a user is for wanting to see that feature, whereas the #1 scale should be more reflective of how it effects vanilla gameplay (being able to disable and dodge traps in adventure mode is something that's needed but might not be highly desired for some users because they primarily play in fort mode, and want to see more realistic mining or farming, whereas for someone else, they might think that realisic farming is not only desirable but necessary for a proper challenge, but also sees the need for adventure mode trap dodging, but is indifferent (a 5 on the 1-10 scale) about when the feature comes).

What the math has to do is find the average location on the XY plot based on the input (each user's input scaled appropriately to a bell curve--if someone rates everything a 8 with a single 2, then on average they've rated things as a 5, except one item as a 1) and give it a score, the higher the score is on both scales would indicate a higher priority on the dev list.

It separates out why people want different features and finds the features that the game needs while supplying features that people desire.

I thought some more about this and it seems to consist mainly of two things: a second dimension (the use of which I find questionable) and the relative order of a person's votes (like a personal ranking of the things). This relative order is already covered in the fractional vote process. So we will have a one-dimensional version of your suggestion at least. It's also what I'm implementing next.

Edit 2: - Up/Down also allows you to see the difference between the sexy stuff people get excited about and the simple engine stuff that everybody just wants to show up.  Perhaps show 'average vote' as a variable in the multiple vote system?

What do you mean with average vote in the multiple vote system? An average of the up/neutral/down vote total per item? I thought I'd better not display this, because it could hurt some people's feelings when they get -345 or something on their suggestion...

[Footkerchief]

An aside from the voting methods -- could you include anchors in the voting page so that people can link directly to a relevant item?  Right now the anchor atts only get inserted for your current votes.

[zagibu]

That's a good idea. It will have to be several links, though, because each method has an own page now.

I also re-read the posts I didn't quite understand before, and I'd like to address two things:

Playing the system is less important here than in e.g. an election, because the output of the voting is a ranking, not a single winner. I doubt Toady will ever get influenced by this ranking so much that he focuses his whole devtime on the top suggestion. Also, what I haven't mentioned yet, the rankings are separate from the voting lists (which are randomly ordered), so playing the system as a side-effect of lazyness should be ruled out.

The neg/neutral/pos system has an important technical difference compared to the rating scale system: You only have to store ups or downs, because you can go with -1, 0 and +1 respectively. If you have a positive-only scale, you either have to store all ratings, or you start the scale with 0, which means extremely disliked will be the default value.

A rating scale system might also be interesting...maye if progress on the other things is faster than expected...

[Footkerchief]

The neg/neutral/pos system has an important technical difference compared to the rating scale system: You only have to store ups or downs, because you can go with -1, 0 and +1 respectively. If you have a positive-only scale, you either have to store all ratings, or you start the scale with 0, which means extremely disliked will be the default value.

Yeah, this comes back to the question of whether or not the final number (votes up minus votes down, presumably) has any absolute meaning, or is just something you use to rank the entries.  Toady was assuming (I think) that it doesn't have absolute meaning.  I'm still not sure.  There's certainly some psychological significance to some people, like in your observation that an entry with a score of 0 seems "extremely disliked."