Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3

Author Topic: DF Memory Per Tile Science  (Read 6074 times)

expwnent

  • Bay Watcher
    • View Profile
DF Memory Per Tile Science
« on: April 17, 2012, 10:46:13 pm »

I don't know if windows task manager is lying to me or what, but here's what I found:

Measuring DF's memory usage before and after I dug out an entire 94x94 z-level of a 2x2 embark, I found that the memory usage went up by over 30 KILOBYTES per tile. Can anyone reproduce this? If this is right it's utterly ridiculous.
« Last Edit: April 17, 2012, 10:50:00 pm by expwnent »
Logged

Loud Whispers

  • Bay Watcher
  • They said we have to aim higher, so we dug deeper.
    • View Profile
    • I APPLAUD YOU SIRRAH
Re: DF Memory Per Tile Science
« Reply #1 on: April 17, 2012, 10:47:49 pm »

inb4 wat

KodKod

  • Bay Watcher
  • Fond of despair and alcoholism.
    • View Profile
Re: DF Memory Per Tile Science
« Reply #2 on: April 17, 2012, 10:47:59 pm »

My brain done thought that this topic was about dwarven homeopathy.
Logged
/人‿‿人\
Tell me what you see. It's a mortal wretched cacophony!
KodBlog: A rage in progress. Updated 20/04/12

runlvlzero

  • Bay Watcher
    • View Profile
Re: DF Memory Per Tile Science
« Reply #3 on: April 17, 2012, 10:50:13 pm »

Magic, magical computer stuff that only the programmer knows about.

It could be holding everything in some kind of weird arcane number matrices which actually speeds up accessing by computing on multiple things at once... just because the maths show that each tile = 30k doesn't mean its not being put to good use. If it uses some database to store stones and pathfind all at the same time because 'dwarves' always know what stone to go to next without it having to be computed for every action... it would explain it some.

I can't explain it in simpler terms, and I lack the knowledge to be all science-y and shit =/ sorry
Logged
I voted for BANANA!

Talvieno

  • Bay Watcher
  • Hello, Death. How's life?
    • View Profile
Re: DF Memory Per Tile Science
« Reply #4 on: April 17, 2012, 11:19:01 pm »

It makes sense. Lots of sense. If the game is holding, in memory, only data for revealed tiles - it explains why larger forts go more slowly. It also explains why forts with many z-levels go slowly. And it also explains why the recent science test done with megasmashing mugs/boulders failed miserably for the most part.

...the rest of you are joking, right?
Logged
Quote from: Mr Frog
Talvieno ... seems to be able to smash out novella-length tales on demand

expwnent

  • Bay Watcher
    • View Profile
Re: DF Memory Per Tile Science
« Reply #5 on: April 17, 2012, 11:37:25 pm »

When I originally posted I edited the text to say something along the lines of "results pending" because hit post before I double-checked the results. The first few are responding to that.
Logged

MagmaMcFry

  • Bay Watcher
  • [EXISTS]
    • View Profile
Re: DF Memory Per Tile Science
« Reply #6 on: April 18, 2012, 01:56:39 am »

DF also holds unrevealed tiles in memory, or how else could you explain Forgotten Beasts and such wandering about in unrevealed parts of the caverns? I'd expect that DF simply has a memory leak. Try reloading your save, I'll bet that DF now uses less memory.
Logged

Talvieno

  • Bay Watcher
  • Hello, Death. How's life?
    • View Profile
Re: DF Memory Per Tile Science
« Reply #7 on: April 18, 2012, 02:09:42 am »

DF also holds unrevealed tiles in memory, or how else could you explain Forgotten Beasts and such wandering about in unrevealed parts of the caverns? I'd expect that DF simply has a memory leak. Try reloading your save, I'll bet that DF now uses less memory.
It's very possible that it can hold them in memory without holding all the data for them. Even closing off areas with forbidden doors doesn't lower the framerate all the way, correct? Perhaps the memory leak has something to do with revealed tiles themselves. Something Toady missed, perhaps.
Logged
Quote from: Mr Frog
Talvieno ... seems to be able to smash out novella-length tales on demand

MagmaMcFry

  • Bay Watcher
  • [EXISTS]
    • View Profile
Re: DF Memory Per Tile Science
« Reply #8 on: April 18, 2012, 02:56:57 am »

I recall that someone ran DF with Valgrind a while ago and found insane amounts of memory leak.
Logged

Uristocrat

  • Bay Watcher
  • Dwarven Railgunner
    • View Profile
    • DF Wiki User Page
Re: DF Memory Per Tile Science
« Reply #9 on: April 18, 2012, 03:31:42 am »

You're forgetting that items & creatures use memory too.  Like the stones you just dug out, wandering animals, or even ambushers sneaking around out there with all their gear.
Logged
You could have berries on the rocks and the dwarves would say it was "berry gneiss."
You should die horribly for this. And I mean that in the nicest possible way.

Talvieno

  • Bay Watcher
  • Hello, Death. How's life?
    • View Profile
Re: DF Memory Per Tile Science
« Reply #10 on: April 18, 2012, 03:59:37 am »

You're forgetting that items & creatures use memory too.  Like the stones you just dug out, wandering animals, or even ambushers sneaking around out there with all their gear.
Forgetting? Not really. You're right, but it's not enough to cause the FPS death that everyone knows and fears.

http://www.bay12forums.com/smf/index.php?topic=104643.210 Proven in this thread, by the venerable NW_Kohaku.

There's something else that does it - and that "something else" could be, oddly enough, walls and empty space. Yes, having to choose which item of 5k to walk to slows it down - but it's intermittent. They only do that once, and then they have a path. FPS death implies that the framerate slowing is constant. I've seen that a forgotten beast with toxic blood causing all my dwarves to vomit everywhere can cause a five second delay every two seconds on my laptop. But for those two seconds in between - everything goes smoothly. Not FPS death. I always figured it wasn't the pathing algorithms that were doing it... somehow I never thought it could be simply rock.

Think about it - for all the tags that a single article of clothing has, you're sitting on a huge embark. I can't remember the exact numbers, and DF is closed, as I'm about to go to bed, so let's go low and pretend for a moment that your average embark is 100 tiles on each side. Now let's pretend that your embark extends downwards 100 z's - you got unlucky and the magma's close to the bottom. That's 1,000,000 slots, and as floors and walls are counted separately - 2,000,000. Every unmined stone needs tags and values - thinking minimum and streamlined (and there may be some I forgot): State (solid, liquid, etc), Material, heat, mined-out percentage, location, and everything that's splashed on it. Assuming that a stone with nothing mined out of it has no tag for that, that's still four values per stone minimum. Seven, if you're counting location as X-Y-Z. This bumps it up to seven to thirteen million values sitting around.

Now of course this is a ridiculously inefficient way of doing things, and Toady, as a programmer, would've realized that from the beginning. Thus, if areas of the map yet to be revealed are simply tagged with a single character: S/L/G (solid/liquid/gas (aka air)) along with, of course, separate values for trees and so on so that you can simulate the cavern growth, you lower it to close to only 1,000,000 characters. You could even use binary: 00/01/10. Nothing's smaller than that. That's far, far faster than using separate tags for everything, and takes up a lot less space. However, if all that expands as far as memory goes when you expose it... it could easily reach around 364,000,000 characters relative to binary. That would explain why everyone says "ohhhhhh, don't breach the caverns unless you want your FPS to go away!!!". There's a reason for it. And it's not just pathfinding AI. I never really thought that was a likely answer to it, anyway, to be honest.
« Last Edit: April 18, 2012, 04:07:00 am by Talvieno »
Logged
Quote from: Mr Frog
Talvieno ... seems to be able to smash out novella-length tales on demand

Uristocrat

  • Bay Watcher
  • Dwarven Railgunner
    • View Profile
    • DF Wiki User Page
Re: DF Memory Per Tile Science
« Reply #11 on: April 18, 2012, 04:31:36 am »

If Toady isn't already familiar with the flyweight pattern, I would hope that he would look into it.  Because you only need to store the raw values for each type of stone once and have each individual stone reference the single copy with the density and other properties.  Though I grant that things in this game have a lot of individualized properties (history, coatings, etc. etc. etc.) that surely causes bloat.
Logged
You could have berries on the rocks and the dwarves would say it was "berry gneiss."
You should die horribly for this. And I mean that in the nicest possible way.

ratchet freak

  • Bay Watcher
    • View Profile
Re: DF Memory Per Tile Science
« Reply #12 on: April 18, 2012, 04:40:29 am »

If Toady isn't already familiar with the flyweight pattern, I would hope that he would look into it.  Because you only need to store the raw values for each type of stone once and have each individual stone reference the single copy with the density and other properties.  Though I grant that things in this game have a lot of individualized properties (history, coatings, etc. etc. etc.) that surely causes bloat.

flyweight still needs a pointer and a deref each time you need the data
Logged

miguelsz2

  • Bay Watcher
    • View Profile
Re: DF Memory Per Tile Science
« Reply #13 on: April 18, 2012, 04:54:16 am »

Spoiler (click to show/hide)

On a serious note... The games in alpha so I would assume that it's fairly normal for performance issues, no?
Logged

SRD

  • Bay Watcher
  • Who the hell do you think I think you are?
    • View Profile
Re: DF Memory Per Tile Science
« Reply #14 on: April 18, 2012, 05:07:20 am »

...

Get DFHack, type into the command reveal and check it then

then type in reveal hell and show the difference.
Logged
Quote from: LoneTophat
EDIT: HOW DO I STOP THE BLEEDING!
SUPEREDIT: Nevermind. Bled to death ._.
Pages: [1] 2 3