Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Dirst]

@Aquillion,

Those sorts of things are usually modeled imperfectly with "decreasing marginal utility."  The first unit of something is the most valuable to a person, the second one a bit less, etc.  Depending on what the thing is, there could be a rapid drop in utility once basic needs are met.  If one has too much of something, the marginal utility of the next unit could easily be negative.

Depending on how complicated we want to make things, a good/service could have a possession utility, a consumption utility, and an experience utility.  That is: having the cake, eating the cake, admiring the cake.  The "consumption" utility for bits of durable furniture is available to tantrumming Dwarves (or invading pillagers) who topple them.  There is a gray area between experience and possession when it comes to assigned equipment... Urist doesn't own the catapult, but she loves operating the weapon that slew the Elven general Rigidwobble the Lantern of Straightcircle.

Exoticness and aesthetic value can be modeled with marginal utility, especially if you consider the first time acquiring/consuming something to be an experience.  Personality traits already have enough breadth to handle one guy collecting art and another just hoarding wealth.

@GavJ,

I agree that fiat currency is not worth shoehorning in early (if at all).  Maslow's Hierarchy of Needs is a good rule of thumb, but it doesn't quite fit onto the microfoundations of behavior (one glaring anomaly is military service which puts Esteem and Love/Belonging needs above Safety and Physiological needs).  It's a good reality check to make sure any system has a reasonably good fit to the Hierarchy, without getting dogmatic about it.

As for market efficiency, once the parties are trading in-person then Pareto efficiency should be the norm among professional traders.  Amateur traders (such as bargaining partners within a fort) put a significant value on "fairness" that appears so against self-interest that economists have re-run the experiment multiple times to convince themselves it actually happens.

Before the traders meet, uncertainty plays an important role.  The caravan loads up things it thinks the other party will want, with an expectation of trading for things it expects the other party will have.  The algorithm each party uses to "expect" things can have a big impact on what actually happens... and as we discussed NPC-to-NPC (algorithm-vs-algorithm) trading will dominate the world economy.  A pessimistic minmax strategy tries to ensure the best possible worst-case scenario, whereas expectations of reciprocity can lead to much better outcomes (see the oddly-named Folk Theorem).

When the trading partner's value is uncertain and sending a caravan is not free, long-distance trade could collapse entirely.  The system will probably need a bit of forcing, such as a norm of sending caravans to all non-hostile neighbors (which is actually an example of the Folk Theorem in action).

[GavJ]

As for market efficiency, once the parties are trading in-person then Pareto efficiency should be the norm among professional traders.  Amateur traders (such as bargaining partners within a fort) put a significant value on "fairness" that appears so against self-interest that economists have re-run the experiment multiple times to convince themselves it actually happens.

I think my examples of unfairness were simply poorly math-ed to demonstrate the problem.  More like:

surplus A   surplus B
1) 50         30
2) 45         45
3) 30         50
4) 5           100

You have room for any 2 of the above four options

[Dirst]

As for market efficiency, once the parties are trading in-person then Pareto efficiency should be the norm among professional traders.  Amateur traders (such as bargaining partners within a fort) put a significant value on "fairness" that appears so against self-interest that economists have re-run the experiment multiple times to convince themselves it actually happens.

I think my examples of unfairness were simply poorly math-ed to demonstrate the problem.  More like:

surplus A   surplus B
1) 50         30
2) 45         45
3) 30         50
4) 5           100

You have room for any 2 of the above four options

That one is trickier to model because all possible combinations are Pareto efficient (you can't make one party better off without making the other worse off).  It comes down to who offers the deals, and the likeliness that they would be accepted.

If A gets to make take-it-or-leave-it offers, the trades will be 1 & 2.  If B gets to make take-it-or-leave-it offers, the trades will be 3 & 4.  If they alternate, it will be 1 & 4.  Without side-payments, the recipient of an offer will always accept it if the surplus is positive.  If side-payments are allowed, they will pick 2 & 4 and split the surplus based on bargaining power.

The current trading system seems to want a specific percentage of surplus, which varies based on skill and the trader's mood.  The same logic above applies to percentages, just imagine that B applies a penalty (required rate of return) in his head before deciding.  For NPC-to-NPC trading, one would hope that the required rates of return would be scaled to (more skilled requirement minus less skilled requirement) and zero.

Edit: corrected what would happen if they alternate.  Turns out it doesn't matter who goes first.

[Aquillion]

@Aquillion
This is definitely of interest for value determining:

Basically, psychological concept that you don't give a crap about things like pretty pictures of your race's military victories, if you're starving to death or living in an open field frequented by bandits.

So in addition to having categories like you are suggesting, some of them should be "grayed out" or perhaps more reasonably, reduced in weight, for game creatures/settlements that are currently under immediate-need quota for basic stuff. So like, if your people are hungry in a settlement, you somewhat undervalue weapons, severely undervalue statues, compared to if you were feeling fed and secure at trade time.

Tangentially, random nitpick: food value here should be purely nutrient value as you wrote it up, since the deliciousness and fanciness of a meal is already covered by subsequent categories of value.

Also, it is important to note that some kinds of value stack together -- such as nutrients and delicious pleasure of food both count at once, or the military effectiveness of a sword and it's aesthetic beauty. But other things don't stack, like something's prestige or symbolic value + its nutrient value (i.e. literally can't have your cake and eat it too)

AND then we also have to consider how all these categories of value change marginally if we want to end up with demand curves for an economics algorithm. They probably all operate differently in this regard.

Good start, though!

Sort of.  The problem is that if the game is going to model that in this much depth, it also has to model the ability of intelligent beings to look ahead a bit and understand which needs are actually pressing.  For example, if I try to trade with a hungry person on the street, no matter how hungry he is at the moment, he should understand things like "I have food at home" or "I can hunt later" and should still be willing to trade for things other than immediate sustenance.

That might seem like a trivial case, but the point is -- Dwarf Fortress isn't going to have actual AI that can compare to a human mind.  At a certain point it has to decide what to abstract out and which shortcuts to take to emulate something that seems like a rational person.  And emulating a hierarchy of needs, if it's not done very carefully, strikes me as something that could result in actors in the game behaving in irrational ways on account of gaps in their ability to interpret their world.

That goes for this whole discussion, too.  You have to remember that the player is to a certain extent a hostile partner going up against whatever fake system the AI uses to emulate value -- they're going to be trying to exploit it and make NPCs accept irrational deals in order to benefit themselves; and if it becomes too easy to do that (because players can spot limitations in the AI's ability to understand its situation, and exploit them) the world will come off as less realistic despite having a more detailed simulation.

I mean, any system like this would probably be an improvement over what we have now -- at the moment, players can literally pick rocks up off the ground and trade them to merchants standing right there -- but it's something to consider.

[GavJ]

I mean, any system like this would probably be an improvement over what we have now -- at the moment, players can literally pick rocks up off the ground and trade them to merchants standing right there -- but it's something to consider.

Well yes, this. "Buy food. Sell same food cooked for 5 minutes and get 20x value."

Anyway, the point of stuff like a hierarchy of needs is not to add unnecessary detail. I think it's necessary. How would you abstract it out? Almost anything short of that would either:
A) need to comrpomise it being a real economy. Such as them magically making food and other necessities as needed when other things were shinier and won out in trading OR
B) need the AI's to cheat in other ways like not needing to eat.

if you have a goal of real production and real consumption of allg oods -- which i think is a pretty good and reasonable basic goal -- then you really do need a hierarchy like this as a fundamental tool. You can ditch the self actualization versus emotional happiness and crap at the top, but you need the base: PHYSIOLOGICAL -> SECURITY -> OTHER STUFF. That's all.

OTHER STUFF here can be represented by other localized, easier to code, less-meta methods.

[Shazbot]

Lets work on a specific problem; three dwarves and a red shirt.

Dwarf A likes the color red and has no shirt.
Dwarf B likes the color blue has no shirt.
Dwarf C likes the color red and has a shirt.

How would each dwarf value the shirt, and who would get the shirt?

As I see it (and all numerical values are for example purposes only):

Dwarf A enters the 'market'. He considers the shirt against his wants and needs and values it based on his situation. He is suffering an unclothed penalty, causing him to lose 100 happiness. He also likes red, and acquiring a red item would grant him a happiness bonus of 50 happiness. Combining the utility (removing an unfulfilled need) and the luxury (meeting a preference), he values the red shirt at 150 dwarfbucks.

Dwarf B enters the market and does the same. The shirt is not his preferred color, but would still remove the unclothed penalty. He values the shirt at 100 dwarfbucks.

Dwarf C enters the market, and values the shirt as a red item at 50 dwarfbucks. However, it is also a utility item he could keep a spare of in his bedroom cabinet, as clothes wear out. Given that he doesn't need a shirt, he is willing to pay half of the utility value, although depending on personality traits like greed he could potentially be willing to pay full price in an item-hoarding behavior. But regardless, he values the shirt at 100 dwarfbucks.

Dwarf A wins the bidding for the red shirt at 150 dwarfbucks. This is desirable for the player because it maximizes dwarf happiness. Dwarf B and C seek out other ways to meet their desires. Dwarf B would ideally seek out a blue shirt while Dwarf C would find a red gemstone, but should no alternatives be found, Dwarf B would bid higher than his happiness value (and have a 'was annoyed by high prices for basic necessities' penalty, perhaps) to acquire a second red shirt.

With a surplus of shirts, Dwarf C begins to buy spare shirts for his cabinet. The fort's cost for a shirt drops below the happiness value ('was pleased at low prices for basic necessities') and we begin to have price signals develop. Shirt production was booming, but there's a lack of pants. Players who track the value of various goods (ideally with these values displayed on the Stocks screen or a new Market screen) can see price signals developing; shirts trading below the value of dyed cloth, pants trading as high as gold bars. Players could then order the production of pants to correct the imbalance.

And here we have some examples of the usefulness of a dynamic value system in the fortress; dwarves (usually) get more of their likes, dwarves (usually) delay hoarding behaviors until after other dwarves have their needs met, and players (usually) identify shortages and surpluses through price signals.

[Dirst]

What you described was

Dwarf A likes the color red and has no shirt.
Dwarf B likes the color blue has no shirt.
Dwarf C likes the color red and has a two red shirts.

Trading away your last shirt imposes the -100 happiness (utils) of not having a shirt.

When individuals are price-takers (no haggling), a rational actor would evaluate all of the feasible trades on an expected-benefit-minus-expected-cost basis (converted to some common units, likely utils) and go with the trade that yields the highest surplus.  Part of the cost is getting to where the trade takes place.

[Scruiser]

Do you guys think simpler, incomplete mechanics that work as placeholders would be helpful for developing the finalized economic system, or that they would slow down progress towards a complete economic system?  By placeholder mechanics, I mean like direct mechanical rules in behavior and simplified heuristics.  By direct mechanical rule, traders would always pay a certain ratio higher for weapons during war time.  By simplified heuristic for example: sites will always make sure they have sufficient food first, even if allowing the poorer demographic to starve would result in higher profits for the upper class actually controlling trading (and thus be more realistic). 

I think this question is relevant to the overall course of our discussion, whether we should try to come up with a simpler system for prices that can be expanded, or to directly work out the implementation for world and fortress economy together.

For short term changes I think Aquillion had the right approach.

In any case, measuring value quickly becomes complicated (outside of supply-and-demand, which is obvious but not enough on its own.)  There are a number of categories it would probably want to consider, weighted by things like cultural values:

Military use
Military equipment you can use has this value; stuff you can't (armor in the wrong sizes, weapons your culture doesn't train with, etc) would normally have reduced value here, although they might still have ascetic or cultural value, or be useful if the civilization has a 'niche' use for them (eg. they might not use a weapon for their normal troops, but a few could still be valuable for special forces or somesuch.)  Material and quality also matter a great deal.  The AI needs a way of determining which equipment is 'best' anyway, for all sorts of other things.

Food value
Most food has to be preserved enough to transport to wherever you intend to sell it, although merchants might want to buy enough to feed themselves for the journey.  Really, this could be divided into "rare spices and exotics" (below) and "raw grain, etc to feed cities that don't produce enough on their own", which are very different values; what I'm referring to here is mostly "ability to keep people from starving to death."  To a certain extent, though, this is covered by supply and demand.

Exoticness
Things you can't produce yourself have a higher value.  This is part of supply and demand, but also recognizes the fact that eg. exotic foodstuffs or curiosities have a natural value to the upper class.  The higher-quality dwarfwork might also be worth more to non-dwarves, since they don't seem to be able to produce craftsmanship of that quality themselves.

Aesthetic value
Aesthetic value gets even more complicated, because different people like different things.  I would categorize it according to spheres and concepts (minimal styles, realistic styles, abstract styles, cultural styles for specific cultures, etc), and have different people and civilizations tend to prefer certain ones, which may change in response to other events.  Obviously quality counts for a lot here.  Materials, instead of having set dwarfbux values, could have ascetic spheres attached, with some (eg. shininess, opulence, etc -- the ones that appear on silver and gold) being weighted to have cultures tend to value them more.  This is the hardest category, because supply and demand don't totally explain it (any high-quality work of art is unique, but people will value them differently, and the reasons aren't easy to explain.)

Symbolic value
Items with histories might have this, especially if the history is relevant to the people considering it.  Additionally, artistic works might have symbolic value if their subject has symbolic value -- eg. elves might place more value on a work of art that shows a sacred tree or a great elvish victory, and less value on one of a great elvish defeat.  (Though it's hard to say -- a great elvish defeat would still be relevant to them and therefore might appeal to certain melancholy or thoughtful elves, whereas a battle totally unrelated to them might be valued lower.)

   Adding separate categories of value to the RAWs, with fixed modifiers to those categories to get final price, would provide the backbone for the AI to start making more complex judgments.  I think metals needs separate RAW tags to indicate to the AI how valuable they are for edged weapons, blunt weapons, rigid armor, and flexible armor, along with an aesthetic value tags.  Same goes for weapons.  A short sword and a long sword may have equal aesthetic value, but the long sword if more valuable as a weapon to races that can wield both one handed.
   There are probably a bunch of cases were good good have RAW tags manually adjusted to help the AI make choices.  The issue is how much will this feature actually contribute to the final economic system.

Anyway, so additional value tags will help game AI make better price/value judements.  The other part of it is:

if you have a goal of real production and real consumption of allg oods -- which i think is a pretty good and reasonable basic goal -- then you really do need a hierarchy like this as a fundamental tool. You can ditch the self actualization versus emotional happiness and crap at the top, but you need the base: PHYSIOLOGICAL -> SECURITY -> OTHER STUFF. That's all.

OTHER STUFF here can be represented by other localized, easier to code, less-meta methods.

I think the separation of trading into categories like that is definitely the right choice short-term, and it has a decent chance of contributing to the final developed system.  Maybe there are other categorizations to consider, while still keeping the same basic priorities.
A few categories to consider:
Critical Short-term: Sustenance food primarily (for all races lacking the NOEAT tag), sustenance drink for dwarfs (and other races that need alcohol), medicine during plagues (when that get implemented),
Critical Capital: Axes (for races that cut wood), Pick (for races that dig out site and/or for races with metalworking), Anvil (for races with metal working), storage for food (food spoils without storage), other must-have tools (as they are implemented), coal/metal for smelting basic quantity of metal, bare minimum furniture (just enough beds for dormitories)
Security: Weapons, Armor, metal/raw material for weapons and armor (use the additional value tags previously discussed to determine how to prioritize, value and price metals in armor and weapons), coal/charcoal for forging weapons/armor, bribes/tribute/money for mercenaries
Non-critical Capital: Hives (beekeeping is nonessential but useful), Jugs(to store honey), mine carts (to make mining more efficient, but not necessary),  extra storage (storage is nonessential for non-food items), nest boxes (priority depends on number of egg layers), fire-proof materials (for kilns, smelters, glass furnaces), coal/charcoal for smelting extra metal, other tools for nonessential industries (as they are implemented)
Commodities: Crafts/toy/instruments (once individual actually use them), nicer quality food, extra food, drinks (for races that don't need alcohol), furniture for individual rooms, other decorations or happiness boosting items that are not necessary (as they are implemented)
Luxury Items: Maximum quality/max value food,  maximum quality furniture and other items, gems/gem decorated items, maximum value furniture and other items

    The AI could first always make sure it has enough short term critical good, then make sure it has critical capital, then invest on security or non-critical capital, then make then maximize value/profit on commodities and luxuries.  The categories also help with other choices.  A rich upper class and poor lower class may lead to focus on luxury items while the lower class starves.  A site aiming for economic development may invest heavily in non critical capital.  A newly established site could focus on critical capital.
    Supply and demand curves take over in the areas of nonessential capital, commodities, and luxuries (either directly implemented, or as an emergent effect).  With critical good, critical capital, and security the curves may look weird/not exist (not implemented or not an emergent outcome).
Thoughts?

Sorry for walls of texts, I've been thinking on this topic for a few days.  Try to just read the sections relevant to your ideas if you don't want to read the whole post.

[Shazbot]

I must have been unclear, the third dwarf already had a shirt in his possession. The market was for a new shirt.

[TerminallyBeard]

TLDR: Quota based system. Barter only system. All sites have complete knowledge of where scarcity/surplus lies and trade surplus items without regard to maximising profit while seeking scarce items without regard to minimising loss. Buy/sell value modified by on-site item scarcity and civ/site profiles (representing need and exoticness), set on site establishment.

Spoiler: What i just said, only longer (click to show/hide)

I think a lot of the knots you're tying yourselves into comes from confrusing mechanisms with simulation. We don't need real economic principles, just the approximation of their effects. For example at the macro end couldn't you just create civ profiles (semi-randomised by site) with quotas of various items/item classes based on population level. Site production and trading basically just tries to maintain these quotas. Add in a couple of hidden quota conditions to simulate quirks/specialisation/current events (eg: maintain double dye stores, premium paid for [material type] item but they don't count toward filling quotas, weapon and armor deterioration rate doubled).
Some items, depending on Civ/Site, could initiate different levels of on-site production when an item's stores reach trigger points above/below the quota. For other items these trigger points instead change a (+/-) multiplier tacked onto buy/sell price and +/- a scarcity/surplus variable (X).
The number of caravans able to be sent by a site per year is determined by wealth. A percentage of caravans, determined by total site scarcity (sum of X), are Site Caravans the rest are Mercantile Caravans. Site Caravans take a randomised selection of surplus goods to a site with a surplus of whatever tops their home-sites scarcity list. They trade for the scarce item at whatever disadvantageous price then initiate 'normal' mercantile trading. Choice of destination could be further weighted by other variables like a trading partner preference or distance (over-ridable at certain scarcity thresholds).
Mercantile caravans simply take a randomised selection of surplus goods to a randomised site weighted by trading partner preference, distance and a demand modifier. No need to be complex, this demand modifier simply looks at the top couple of items (by dwarfbuck value) the merchant is carrying and rates each potential destination according to whether these items are scarce.

eg: most valuable items caravn has loaded are cloth, crafts and mugs
destination 1 : needs none of these - demand modifer = 1
destination 2 : only cloth is scarce - demand modifier = 2
destination 3 : cloth, crafts and mugs are scarce - demand modifier = 4

So, its basic, its been suggested a gazzilion times... can anyone explain (to a non-economist) why this wouldnt work? Apologies if I just tracked muddy boots through the ivory tower but I believe the best approach is simply to get those commodities moving and see what emerges.

[Scruiser]

So, its basic, its been suggested a gazzilion times... can anyone explain (to a non-economist) why this wouldnt work? Apologies if I just tracked muddy boots through the ivory tower but I believe the best approach is simply to get those commodities moving and see what emerges.

    One particular egregious problem your quota system has is the lack of profit maximization would result in no way of modeling monopolies, war profiteering, price gouging during famine (and other global shortages), and similar cases.
   In general, I think simple systems like your suggestion could probably work for a placeholder, but you have to add enough inelegant rules for edge cases that it eventually becomes simpler to just get design a complete, holistic economic system to begin with.
   In your example, I think it would have several issues that to completely cover, you would need to add in enough rules to make the quotas almost irrelevant.  Your system has no way of prioritizing classes of goods (food over weapons/armor, and weapons/armor over luxury items for example).  So lets say you add that feature with priority lists for categories of goods.  You still don't have a way of modeling weird demographics (a city has rich members that prioritize luxury items over food for the poor underclass.)  You still need to add in separate quota lists for sites in separate stage of development (a site just being settled will need to have lots of tools/capital that an established site has already in stock).  So you add in a running count of inventories to your quota system.  But that still doesn't account for sites that may need more tools than other (a fortress established for mining need more picks versus a military outpost needs more weapons/armor).  If you kept tacking on rules to cover these cases without having planned on adding them, it makes the whole economic system messy and/or buggy.  If you plan to add all these features and more in the future as you set up your quota system, then you pretty much have to figure out everything this thread is planning for.

TLDR; basically simple systems like yours could work for placeholders, but in the long run you need a complete plan, otherwise your simple system will get cluttered with more and more rules to cover everything.

That said, I think a quota system might be a good start, followed by a system for categorization and prioritization like I suggest (Critical short term, critical capital, security, non-critical capital, commodity, luxury), followed by even more nuanced elements (site leaders influencing priorities, fashion trends, cultural/civ modifiers, creature personality modifiers, etc.).

[GavJ]

...and trade surplus items without regard to maximising profit while seeking scarce items without regard to minimising loss.,,

This would screw up the whole economic system being proposed here. Everything should always be decided based on profit (i.e. maximal value, not money). Otherwise you're undermining the economic assumption. The reason for the economic assumption is that actual economics is what works in a system where everybody is intelligent and selfish. When you deviate from that, the player of the game (who unlike your shortcut program is maximally deviant and selfish) will be able to exploit the hell out of it, making it less fun.

In the example of quotas, the exploits are pretty easy: just buy up all their "surplus" junk beyond their quota and get unfairly and unrealistically low prices for it, then go sell it for the profits they SHOULD have gotten if they actually asked a proper economic price for it instead of arbitrarily falling off a value cliff once they hit their quota. The player just gets to soak up a bunch of excess profits he doesn't deserve, which makes it less realistic, less challenging, and thus less fun.

There ARE some things that physics would provide real underlying numbers for. Like a dwarf actually needs X many calories a year to survive. That's not arbitrary, that's biology. But although this will affect value curves, it would never act like a sharp cutoff quota where they stop valuing any food worth anything once they're full. That's just like a big "rob me blind" flashing neon sign for the player if they do that.

[Scruiser]

In the example of quotas, the exploits are pretty easy: just buy up all their "surplus" junk beyond their quota and get unfairly and unrealistically low prices for it, then go sell it for the profits they SHOULD have gotten if they actually asked a proper economic price for it instead of arbitrarily falling off a value cliff once they hit their quota. The player just gets to soak up a bunch of excess profits he doesn't deserve, which makes it less realistic, less challenging, and thus less fun.

This didn't occur to me, but exploit are probably one of the primary problems with any simplistic placeholder system.  Whatever the system fails to take into account, you should assume that the players will take advantage of it.  For any hard and fixed rules, you should expect the player to find a way to play the rules against each other.  For any decent economic system, there should be some type of generalized mechanics that will stop exploits in cases that aren't directly covered (i.e. supply and demand providing effective limits on quantity and prices) .

[GavJ]

I don't mind intermediate steps, even if exploitable, because theyd still be less exploitable than things are now.

I'm only objecting to what appears to be a suggestion for a FINAL system that simplifies in an easily exploitable way. For something like quotas, it doesn't seem like it's a step toward anything, it just seems like an endpoint. In other words, if you did a full fledged econ system later, you'd just have to rewrite it, meaning the effort spent making the earlier system is wasted.

Unless I misunderstand and it is meant as a stepping stone somehow.

[c6r]

TLDR: Quota based system. Barter only system. All sites have complete knowledge of where scarcity/surplus lies and trade surplus items without regard to maximising profit while seeking scarce items without regard to minimising loss. Buy/sell value modified by on-site item scarcity and civ/site profiles (representing need and exoticness), set on site establishment.

Spoiler: What i just said, only longer (click to show/hide)

I think a lot of the knots you're tying yourselves into comes from confrusing mechanisms with simulation. We don't need real economic principles, just the approximation of their effects. For example at the macro end couldn't you just create civ profiles (semi-randomised by site) with quotas of various items/item classes based on population level. Site production and trading basically just tries to maintain these quotas. Add in a couple of hidden quota conditions to simulate quirks/specialisation/current events (eg: maintain double dye stores, premium paid for [material type] item but they don't count toward filling quotas, weapon and armor deterioration rate doubled).
Some items, depending on Civ/Site, could initiate different levels of on-site production when an item's stores reach trigger points above/below the quota. For other items these trigger points instead change a (+/-) multiplier tacked onto buy/sell price and +/- a scarcity/surplus variable (X).
The number of caravans able to be sent by a site per year is determined by wealth. A percentage of caravans, determined by total site scarcity (sum of X), are Site Caravans the rest are Mercantile Caravans. Site Caravans take a randomised selection of surplus goods to a site with a surplus of whatever tops their home-sites scarcity list. They trade for the scarce item at whatever disadvantageous price then initiate 'normal' mercantile trading. Choice of destination could be further weighted by other variables like a trading partner preference or distance (over-ridable at certain scarcity thresholds).
Mercantile caravans simply take a randomised selection of surplus goods to a randomised site weighted by trading partner preference, distance and a demand modifier. No need to be complex, this demand modifier simply looks at the top couple of items (by dwarfbuck value) the merchant is carrying and rates each potential destination according to whether these items are scarce.

eg: most valuable items caravn has loaded are cloth, crafts and mugs
destination 1 : needs none of these - demand modifer = 1
destination 2 : only cloth is scarce - demand modifier = 2
destination 3 : cloth, crafts and mugs are scarce - demand modifier = 4

So, its basic, its been suggested a gazzilion times... can anyone explain (to a non-economist) why this wouldnt work? Apologies if I just tracked muddy boots through the ivory tower but I believe the best approach is simply to get those commodities moving and see what emerges.

Any system that utilizes a hard coded intermediary unit of value (IE Dorfbucks) is going to be broken and exploitable.  Getting rid of that exact thing should be the centerpoint of any properly modelled functional economy, in my opinion.  Honestly, in my mind, if the microeconomy of the individual fort is properly modelled, the trade with external civilizations is easy to work.  Its a simple trading post economy and, really, can be mostly automated.  The player can set the money good (which, lets face it, 99.99999% of the time its going to be local stone because it has utility value to every civilization, but none of them are capable of producing it in quantity like dorfs, for whom its basically just a waste by-product of doing something of actual value... digging), set temporary mandates and export bans to influence the supply and demand curves to make it more likely to trade for a particular good, and assign special items to trade like specific finished goods (basically like we do now)... but for the most part, as long as the trading post is open, the broker is going to automatically export the money good (and whatever else has been assigned to the post) for whatever the trade caravan has based on the negotiation skills of both parties and the aggregate supply/demand curves of their respective civilization.  Live on a mountain and have an infiltrate supply of granite, but no wood?  Expect that when a trade caravan from the Elves shows up (provided that they are looking for rock... which they should if they are sending a caravan in the first place.  The AI will have awareness of what the primary trade good is at a particular fort's trade post), they are going to take a lot of granite away and dump a lot of wood into your local economy.  Obviously, there is lot of stuff that would be going on under the hood, but, generally, from the player's perspective, they would just say whether trading happens at all and set the basic criteria and goals for trading, but wouldn't otherwise influence the individual trades themselves.  I feel that is consistent with the spirit of Dwarf Fortress... direction, but not "control", per se... "managed chaos".

I don't know... just my thoughts.