Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3

Author Topic: Route caching  (Read 5900 times)

Ogantai

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #15 on: December 04, 2008, 06:13:48 am »

path caching: http://www.bay12games.com/forum/index.php?topic=24790.0
I might have missed it but I didn't see it mentioned, how much memory did your route cache take up? Was it less than the amount saved by reducing route calculations?
Logged

sir monko

  • Bay Watcher
    • View Profile
    • wsa
Re: Route caching
« Reply #16 on: December 04, 2008, 11:30:15 am »

@how much memory did your route cache take up?

i can't really say - it depends on the implementation (i don't have any insights into the dwarf fortress code itself). do you store vectors or directions? how complex do they have to be? can you use parts of paths? you could even use compression ... it's all mentioned in the other thread. if you get down to the basics, you could use an array of bytes. 1 byte per step. then RLE it. i'd say, a path of 100 steps would go down to the half. plus the overhead of the linked list or binary tree or whatever stores the paths.
my opinion: cache size doesn't really matter. you could cache thousands of them, but in the age of gigabytes of ram, 2mb more or less don't really matter.

you could even assign a certain amount of memory to cache, and if cache's full, run the cache-GC and/or just don't cache more paths. you could turn the cache off, cpu load would go up, memory usage down, otherwise nothing different.


@performance:
i did a very simple demo in java, but it didn't use A* (too lazy), so i can't provide real benchmarking results.

in my simple and small 2d demo maps, the cache served up to 85% of the requested paths. another, more complex map gets down to 60% cache hits (Targets selected: 4515, Total paths requested: 22644, Total paths calculated: 9069, Ratio: 40.050346). even these numbers you could heavily tweak by changing the GC-parameters (eg. max age of cached paths).

in fact, i'm pretty sure it would pay off, at least inside the fortress. A* is performant, but i'm sure the overhead of a (failed) cache lookup is not that bad.

@ trombonistAndrew:
ant colony optimization is not very useful here. people like it, because the dorfs look a lot like ants, but they act differently. and TSP is, as you mentioned, a totally different problem.

genetically based algorithms could probably used for tweaking the A* parameters (different fortress layout, different optimal paramters), but i doubt there is a possibility to use them for path finding directly.

my opinion: path cache is very easy to implement and has the potential for severe performance gains. A* itself is performant enough, leave it as it is.
a caching engine would affect the gameplay a little bit (dwarves running into walls a bit more often), but could be implemented in a way it could easily turned off and limited to a certain amount of memory.
Logged

Ogantai

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #17 on: December 04, 2008, 05:09:45 pm »

A* is performant, but i'm sure the overhead of a (failed) cache lookup is not that bad.
Right, IIRC it should be O(n) at worst.
Logged

sir monko

  • Bay Watcher
    • View Profile
    • wsa
Re: Route caching
« Reply #18 on: December 04, 2008, 08:26:32 pm »

A* is performant, but i'm sure the overhead of a (failed) cache lookup is not that bad.
Right, IIRC it should be O(n) at worst.
depends on the data-structure you're using. looking for A->B (and B->A), where A and B are pairs of 3D coordinates, you could use some kind of tree or hashing to lower the lookup costs even more.
Logged

scribbler

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #19 on: December 05, 2008, 10:44:48 am »

One last  ::) idea from the less mathematical department. These sorts of issues, in human history, are caused by and solved by people. Why not make the dwarves responsible?
Have a sign as a piece of furniture or inscription and use those as way points, caching just between them. We can put down whatever we think should be key points, funneling traffic through lights and dwarf washes and away from the siege, and the dwarves would favor those, just like they favor high traffic areas or we favor certain roads when it's not always the fastest way.
The dwarves check for the nearest sign to them and their destination, to minimize time a quick distance check plus the known route length is compared to a direct line to their destination and take which ever seems shorter. Yes, they'd make poor choices, that's where the fortress designer comes in. Hell, they can have bad thoughts from not having marked routes. (He is upset about that it takes longer than it should to get to the dining room. (He had to take the long way around and wants a route marker to fix the check).)
I've already taken to leaving a pillar in the center of intersections as I build. It looks pretty cool and seems more dwarvish than fountains and statues.
Logged
End the slaughter of dorf kittens!
No self respecting beard wants to wear his pet as clothing! Dorfs need population control for pets and bad thoughts from products made from the animals they choose to bond with.
---------
"There are two means of refuge from the miseries of life: music and cats."
-Albert Schweitzer

Andir

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #20 on: December 05, 2008, 03:07:26 pm »

These sorts of issues, in human history, are caused by and solved by people. Why not make the dwarfs responsible?
On average though, people tend to have memory of where they've been, a sense of direction, and a sense of distance to be able to calculate the next logical step.  You can also look out into the horizon to see if there's a wall to go around and deduce/remember which way the door is.  Dealing with a simulation like this, each dwarf has very little memory to remember routes like humans would.  They rely on the engine to tell them where to go.  The problem with markers is when you have none.  The dwarfs would have to be able to navigate a little.  If you can get them to nav a little, why not use that instead of engine tricks.

If you had a signpost mechanism that was updated every time something was built near it, then the dwarf would only need to find the closest sign to get directions.  However, what kind of directions?  Go west for workshops?  What if you have workshops both ways?  You'd have to recursively update each sign with directions on getting to the shop you just built.  That means that every sign must be aware of that shop you just built and be able to tell a dwarf which sign to go to next.

For instance:
You build signs at every intersection.
You build another sign, workshop, designate a room, or storage.
The build function finds the closest sign and updates it with a reverse route to the sign/shop/room/storage.
That sign would have to update it's neighbors, and so on until every sign knew the location of that new shop.

That's quite a bit of memory if you use the signs.  If you don't use the signs, then how do the dwarfs know their way around?  People don't need signs to get somewhere.  Usually through trial and error you find your way.  They also use markers to help them.  So why not have the dwarfs drop markers (virtually) after building a shop?  When they finish building, they route the path back to the entrance and lay down markers?   How do your dwarfs know it's an intersection and not a very big room, wide path or a turn?

You could make a new skill called fortress cartography and have cartographers scout around your fortress looking for new mine shafts and update the maps in the great meeting hall.  ;D

It's a fun sort of problem though. ;)
Logged
"Having faith" that the bridge will not fall, implies that the bridge itself isn't that trustworthy. It's not that different from "I pray that the bridge will hold my weight."

Sowelu

  • Bay Watcher
  • I am offishially a penguin.
    • View Profile
Re: Route caching
« Reply #21 on: December 05, 2008, 09:29:45 pm »

The ant-based approximation to the Travelling Salesman thing is designed for cases where CPU and memory are very cheap compared to the cost of driving around the country for days on end.  I -can- see a huge advantage to route caching though...  My main quibble is that route caching could deal somewhat well with 'my route is blocked' (just find a new one), only moderately well with 'my route has heavy traffic, or even just light traffic', and couldn't deal at all with "someone just dug a new staircase that saves me two hundred squares of movement".

While it would be really cool to try and figure out where 'rooms' are for pathing, it might be more realistic to just path from every building to every other building, where stockpiles count as buildings.  Or heck, even ignore pathing from a mason's workshop to a clothier's shop.  That would give good distance estimates for like 75% of the items lying around your average fort, which is something at least.  It would still suck for stone.
Logged
Some things were made for one thing, for me / that one thing is the sea~
His servers are going to be powered by goat blood and moonlight.
Oh, a biomass/24 hour solar facility. How green!

Footkerchief

  • Bay Watcher
  • The Juffo-Wup is strong in this place.
    • View Profile
Re: Route caching
« Reply #22 on: December 06, 2008, 02:37:25 am »

The ant-based approximation to the Travelling Salesman thing is designed for cases where CPU and memory are very cheap compared to the cost of driving around the country for days on end.  I -can- see a huge advantage to route caching though...  My main quibble is that route caching could deal somewhat well with 'my route is blocked' (just find a new one), only moderately well with 'my route has heavy traffic, or even just light traffic', and couldn't deal at all with "someone just dug a new staircase that saves me two hundred squares of movement".

I don't see the staircase scenario as a problem per se.  That's exactly the kind of "flawed" pathfinding exhibited by real humans.
Logged

Draco18s

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #23 on: December 06, 2008, 12:39:06 pm »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.
Logged

scribbler

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #24 on: December 06, 2008, 11:04:15 pm »

On average though, people tend to have memory of where they've been, a sense of direction, and a sense of distance to be able to calculate the next logical step.  You can also look out into the horizon to see if there's a wall to go around and deduce/remember which way the door is.
I find myself having to give directions on a regular basis. Perhaps the players of Dwarf Fortress are a cut above the rest, but most people are baffled by things like the name of the mall where they shop, the location of the only major roadway and the knowledge of where west is. And, having recently moved to an area with lots of twisting roads and no intersections I'm always baffled by the direction the sun sets at my job compared to my house. Now, do that underground in twisty little passages all alike. (Which I've also done.)

Dealing with a simulation like this, each dwarf has very little memory to remember routes like humans would.  They rely on the engine to tell them where to go.  The problem with markers is when you have none.  The dwarfs would have to be able to navigate a little.  If you can get them to nav a little, why not use that instead of engine tricks.
Because a) it takes extra memory b) by pathfinding common or difficult areas in advance we save all that repitition.

If you had a signpost mechanism that was updated every time something was built near it, then the dwarf would only need to find the closest sign to get directions.  However, what kind of directions?  Go west for workshops?  What if you have workshops both ways?  You'd have to recursively update each sign with directions on getting to the shop you just built.  That means that every sign must be aware of that shop you just built and be able to tell a dwarf which sign to go to next.
No, you'd find the nearest sign to the dwarf and the nearest to the target and do a quick check to see if it's worth using the main route. When the route's off or not accessible then regular pathfinding takes over.
It's merely a hint to the computer so that it can save a few cycles without caching 4,000 routes and to control paths where geography might make it awkward.

For instance:
You build signs at every intersection.
You build another sign, workshop, designate a room, or storage.
The build function finds the closest sign and updates it with a reverse route to the sign/shop/room/storage.
That sign would have to update it's neighbors, and so on until every sign knew the location of that new shop.
a) Signs wouldn't know shops, only other signs b) Signs only update when there's time or as a byproduct or regular dwarf pathing. I'm sure there's some sort of neural net type terminology for the signs knowing each other and finding a route primarily through the nodes.

That's quite a bit of memory if you use the signs.  If you don't use the signs, then how do the dwarfs know their way around?  People don't need signs to get somewhere.  Usually through trial and error you find your way.  They also use markers to help them.
Actually, we have signs everywhere. Walk, don't walk, Main Street, Route 12, 23-86 South St., 6th Floor, Rooms 631-640 right, Suite 639: Ed's Tackle Inc., Ms. McReceptionist, Receptionist, Up, Down, Slippery When Wet. Large cities have signs EVERYWHERE (not counting advertising). Rural areas have signs all over the highway and filling every rest stop. And then we still have them on our mailboxes, cars, streets, schools, offices... The markers we mostly use are signs. Turn left at McDorf's, (the place with the big gabbro M sign). Even so, a statue or fountain could be just as effective a landmark, I'm merely using the term sign, perhaps station is more appropriate, but it evokes images of dwarves using minecarts for mass transit.

So why not have the dwarfs drop markers (virtually) after building a shop?  When they finish building, they route the path back to the entrance and lay down markers?   How do your dwarfs know it's an intersection and not a very big room, wide path or a turn?
They don't. They're dwarves. They charge naked to drink from carp rivers and let their puppies sleep on traps. And dropping all those markers goes back to the whole huge routing thing.

You could make a new skill called fortress cartography and have cartographers scout around your fortress looking for new mine shafts and update the maps in the great meeting hall.  ;D
It's an alternative to the dead time solution. (I really would like DF to use the time while I scout and plan and examine and trade and use the facilities. It would be great if I could pause it to get some water and come back to find that everyone has reevaluated their tasks and routes and selected the best materials and routes.


It's a fun sort of problem though. ;)
That it is.

Just to clarify, signposts shouldn't be dropped at every intersection, that would make them street signs, good for RP, but maybe not for a system. More likely a signpost by the dining room, workshops, a back staircase which is being ignored, the preferred stockpiles...
McUrist is 103 squares from his bed.
McUrist is 23 squares from a sign and his bed is 14 squares from a sign, the path between is already done at 88 steps. (37 squares and 88 steps.)
Now 103 is less than 125 (37+88) but the 125 is optimised and doesn't need to be pathed except for the ends. (McUrist might also be entitled to a certain number of frequently used paths based on his intelligence (or cartography skill ;)), thus making smarter people potentially better performers. It might also be guarded, legendary and the 103 is a rough or straight line quick calculation. He'd likely take more or less the same route in the end only he might take a less effective route and take more time calculating it.
Now, McScrewed is 118 from a sign heading someplace 39 squares away (110 squares from a sign), but! McScrewed after pathing his little heart out flashes a ? while giant cave spiders eat him because the tunnels were all convoluted and the route he found with his dying breath is only 20 squares shorter. He should have taken the safe route and saved himself the ? time in the process.

Of course there could be a couple of options in the raws to tweak the sign functions. By raising a square length you can make known routes vastly favored over unknown and even drive dwarves to seek out established paths or choose stones nearest the "highway." Or by turning it down, the routes are little more than a hint.

Maybe someone can make up a basic map of an area in their own fortress with pathing difficulties we can use to discuss ideas?
Logged
End the slaughter of dorf kittens!
No self respecting beard wants to wear his pet as clothing! Dorfs need population control for pets and bad thoughts from products made from the animals they choose to bond with.
---------
"There are two means of refuge from the miseries of life: music and cats."
-Albert Schweitzer

Osmosis Jones

  • Bay Watcher
  • Now with 100% more rotation!
    • View Profile
Re: Route caching
« Reply #25 on: December 07, 2008, 12:45:07 am »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.

Didn't Sir Monko (the guy in the other caching thread) compensate for this by discarding older routes? Sure, it increased the processor load somewhat, but it allowed for the updating of routes over time.
Logged
The Marx generator will produce Engels-waves which should allow the inherently unstable isotope of Leninium to undergo a rapid Stalinisation in mere trockoseconds.

Draco18s

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #26 on: December 07, 2008, 03:16:07 pm »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.

Didn't Sir Monko (the guy in the other caching thread) compensate for this by discarding older routes? Sure, it increased the processor load somewhat, but it allowed for the updating of routes over time.

I don't remember.
Logged

Footkerchief

  • Bay Watcher
  • The Juffo-Wup is strong in this place.
    • View Profile
Re: Route caching
« Reply #27 on: December 07, 2008, 03:36:36 pm »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.

Didn't Sir Monko (the guy in the other caching thread) compensate for this by discarding older routes? Sure, it increased the processor load somewhat, but it allowed for the updating of routes over time.

I don't remember.

Aging is easy in any case.  And it's not hard to directly detect a possible need for new routes -- like if you're caching A -> B and B -> C, and the game notices that the two routes are often used together, it can make a new shortcut path for A -> C.  If dwarves then start pathing from C to a newly accessible point D, the game caches C -> D, and shortly after may cache A -> D.  So new paths can develop organically.

Anyway, I hope Toady has noticed these threads, particularly sir monko's.  The performance boosts from this kind of thing could be pretty significant, and as sir monko pointed out in the last thread, the cache size could easily be dynamically adjusted to balance performance with memory usage.
« Last Edit: December 07, 2008, 03:38:12 pm by Footkerchief »
Logged

Ogantai

  • Bay Watcher
    • View Profile
Re: Route caching
« Reply #28 on: December 07, 2008, 08:02:23 pm »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.

Didn't Sir Monko (the guy in the other caching thread) compensate for this by discarding older routes? Sure, it increased the processor load somewhat, but it allowed for the updating of routes over time.

I don't remember.
If all else fails just give users an option to flush the route cache manually, problem solved.
Logged

profit

  • Bay Watcher
  • Finely Crafted Engravings... Or it didn't happen.
    • View Profile
Re: Route caching
« Reply #29 on: December 08, 2008, 10:47:17 pm »

Except that in a simulation there is no way of alterting the cache system that there is a new route, unless you did it EVERY time a tile was dug out.  There simply isn't a way to find new paths.

Didn't Sir Monko (the guy in the other caching thread) compensate for this by discarding older routes? Sure, it increased the processor load somewhat, but it allowed for the updating of routes over time.

I don't remember.
If all else fails just give users an option to flush the route cache manually, problem solved.

Or just every 8 uses of the cached route, discard and recaclculate.  We could have 40 dwarves instead of 5 before it lags in that case =p

* actually the real lag is caused by items that dwarves are unable to reach and too big of areas being designated.  Pathfinding is harsh but currently caching it is not the only way of optimizing it.  There is alot more messy code that can be cleaned up.
Logged
Mods and the best utilities for dwarf fortress
Community Mods and utilities thread.
Pages: 1 [2] 3