Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 296 297 [298] 299 300 ... 796

Author Topic: if self.isCoder(): post() #Programming Thread  (Read 906606 times)

alway

  • Bay Watcher
  • 🏳️‍⚧️
    • View Profile
Logged

dreadmullet

  • Bay Watcher
  • Inadequate Comedian
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4456 on: June 11, 2013, 01:12:14 am »

http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx
How virtual memory actually works.

Why does that article seem to suggest that everything that gets put into memory also gets put into storage? That's not actually the case, is it?
Logged

Siquo

  • Bay Watcher
  • Procedurally generated
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4457 on: June 11, 2013, 02:27:58 am »

http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx
How virtual memory actually works.
Yeah no, this is more a "How I imagine memory to work" by the author. He's correct (and pedantic) that "out of memory" does not mean physical memory but that the (contiguous) block you reserved can't be allocated, but the rest... Assuming everythis is virtual memory and real memory is merely a smart "cache" is... well I can imagine that if you write memory-heavy stuff on low-end machines that the analogy works, but if you do really CPU intensive stuff for instance, it's very useful to know/control (and you can with trickses) what is in the CPU cache, what is in RAM, and what is on disk.
Logged

This one thread is mine. MIIIIINE!!! And it will remain a happy, friendly, encouraging place, whether you lot like it or not. 
will rena,eme sique to sique sxds-- siquo if sucessufil
(cant spel siqou a. every speling looks wroing (hate this))

Mego

  • Bay Watcher
  • [PREFSTRING:MADNESS]
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4458 on: June 11, 2013, 03:11:35 am »

http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx
How virtual memory actually works.

Why does that article seem to suggest that everything that gets put into memory also gets put into storage? That's not actually the case, is it?

How Memory Actually Works (From A Hardware Perspective)

Generally speaking, there are 4 main categories of memory. First, there is the register memory inside of the CPU. Modern CPU's have several high-speed registers for temporarily storing data. They're expensive, so there's not a whole lot of them. For example, the i386 has 16 32-bit registers.

The next category of memory is cache memory. Cache memory exists to speed up memory access by prefetching data from main memory, so that, when the data is needed, it can be read from the much-faster cache instead. Cache memory is less expensive than register memory, but is still pretty expensive compared to main memory. Modern computers usually don't have more than a few megabytes of cache memory. In addition, modern computers also take advantage of multilevel caches - caches that are made up of different levels with different speeds. The L1 cache is faster (and therefore more expensive) than the L2 cache. Some computers even have a third level, the L3, which is slower and cheaper than L2. Multilevel caches allow for more data to be stored in the cache without significant additional cost, lessening the need to read from main memory.

The third category is main memory. This is what most people think of as computer memory. Nowadays, it comes on rectangular circuit boards that you (or somebody else) sticks into the memory ports on your computer's motherboard. It's pretty cheap, compared to other types of memory, but it's also relatively sluggish. Not only is access time much higher than register and cache memory, accessing main memory also requires using the system bus, which connects all external components (main memory, hard drive, external storage devices, keyboard, mouse, monitor, webcam, speakers, microphone, etc.) to the CPU. Because of that, often the CPU will have to wait until the system bus becomes free in order to access main memory, which equates to more delay between the request being made and the data being transferred. As you can see, reading from main memory isn't ideal. That's why registers and caches are used; to circumvent the long access times for main memory.

Now, with those three categories of memory, it may seem like you're set. But, consider the fact that those three categories only add up to a few gigabytes of memory. A poorly-written Java program could quickly eat all of that up. So what if we need more memory? There's a physical limit on how much main memory a motherboard can hold. Well, we go to another storage device: the hard drive. Unused space on a hard drive can be used for virtual memory. Hard drives are cheap, massive storage devices; modern drives are capable of reaching the terabyte ranges for under $100. The downside? Hard drives give a whole new meaning to the word slow. Not only do you have to deal with the system bus; you also have to factor in the fact that the drive is spinning, and you have to wait until the right locations pass under the drive heads to pull the data. However, when quantity is the main issue, virtual memory is the solution.

So to answer dreadmullet's question: sort of. You're implying that there is a substantial difference between system memory and data on a drive. There really isn't. The only difference is how that data is being used, and the hardware doesn't know the difference, and doesn't care, either. To answer what I think you meant, no, not everything that gets stored in physical memory also gets stored in virtual memory, unless you're on a computer that treats virtual memory as main memory, and main memory as more of a separate cache for the virtual memory.

If you're interested in more details, research mapping schemes (cache <-> main memory, main memory <-> virtual memory), page tables, the translation lookaside buffer, instruction pipelining, and page files.

dreadmullet

  • Bay Watcher
  • Inadequate Comedian
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4459 on: June 11, 2013, 06:11:29 am »

Excellent overview of memory! But I'm still confused about parts of the article. It's these parts I was confused about:

Quote
This is the key point: the data storage that we call “process memory” is in my opinion best visualized as a massive file on disk.
...
...
...
But I much prefer to think of storage as being a chunk of disk storage, and physical memory being a smart caching mechanism that makes the disk look faster. Maybe I’m crazy, but that helps me understand it better.

Also, you didn't mention hard drive caches for whatever reason. I remember picking out a hard drive because it had a higher cache size than comparable drives. I have no idea how important it is, though. But I do remember testing it, and a ~500kB file that loaded in about 0.25 seconds the first time loaded in 0.001 seconds the second time. Or would memory caching come into play there? I have no idea.
Logged

Virex

  • Bay Watcher
  • Subjects interest attracted. Annalyses pending...
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4460 on: June 11, 2013, 09:05:46 am »



The next category of memory is cache memory. Cache memory exists to speed up memory access by prefetching data from main memory, so that, when the data is needed, it can be read from the much-faster cache instead. Cache memory is less expensive than register memory, but is still pretty expensive compared to main memory.


Doesn't the stack also go in cache memory? The X86 chipset doesn't have a separate memory chip for the stack as far as I know, so it should either go in cache or in main memory and since the latter is slower that seems like an odd choice.
« Last Edit: June 11, 2013, 09:12:55 am by Virex »
Logged

Mephisto

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4461 on: June 11, 2013, 12:07:44 pm »

[storytime]

I do web development (I refuse to use the term "full stack" though it would apply). Our newest product, of which I am a major contributor, is a web site that assists people with a certain task. Said site is powered by Django and MySQL.

Our code generally smells pretty good, but one of my most recent tasks was focused on improving a slight smell created by another developer. It was a simple change and went exactly as smoothly as it should have. It worked fine in my local sqlite db. It worked fine in the tester's MySQL db. On staging, however, it dropped a fairly important table. Apparently our staging server uses the production db? >.< Suddenly, no more new signups.

What.

The only migration in my branch consisted of adding a single datetime field to one model. Somehow, the addition of this simple, non-vital field causes a foreign key violation. Upon such, MySQL or South drops the table instead of stopping right then.

I repeat. What.

[/storytime]

What have we learned from this? Even though a change works on every developer's machine, some of which are damned near identical (meaning: version numbers of all software involved match and all computers run Ubuntu 12.x-13.x) to our staging/production servers, said change may destroy everything you know and/or love when deployed to production.

Luckily, the data lost is easily recreated. I'm still going to look into potentially standardizing development environments to hopefully prevent a more catastrophic loss in the future, though.
Logged

kaenneth

  • Bay Watcher
  • Catching fish
    • View Profile
    • Terrible Web Site
Re: if self.isCoder(): post() #Programming Thread
« Reply #4462 on: June 11, 2013, 01:38:51 pm »

Web server Staging running off the Prod backend is normal and expected.
Logged
Quote from: Karnewarrior
Jeeze. Any time I want to be sigged I may as well just post in this thread.
Quote from: Darvi
That is an application of trigonometry that never occurred to me.
Quote from: PTTG??
I'm getting cake.
Don't tell anyone that you can see their shadows. If they hear you telling anyone, if you let them know that you know of them, they will get you.

GlyphGryph

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4463 on: June 11, 2013, 01:50:34 pm »

Web server Staging running off the Prod backend is normal and expected.

Not anywhere I've ever worked. And for damn good reason. Maybe if you're working of something unimportant and trivial...
Logged

kaenneth

  • Bay Watcher
  • Catching fish
    • View Profile
    • Terrible Web Site
Re: if self.isCoder(): post() #Programming Thread
« Reply #4464 on: June 11, 2013, 03:06:45 pm »

Only a billion+ dollar annual revenue website... but there are many levels of test environment before production staging; production staging is where a few select test cases are run that can only be done for 'real', like test purchases using real credit cards, just before public web traffic is directed to the new version; it is, for all purposes the real environment, but with no public DNS ponting to it (using hosts files and browser site certificate checking turned off)

Prod
Prod Staging
Pre-Prod
Pre-Prod Staging
Test
Test Staging
Pre-Test
Pre-Test Staging

before that there are many smaller test environments provisioned on VMs for testing code, content, localization, etc. that don't use the staging process.
Logged
Quote from: Karnewarrior
Jeeze. Any time I want to be sigged I may as well just post in this thread.
Quote from: Darvi
That is an application of trigonometry that never occurred to me.
Quote from: PTTG??
I'm getting cake.
Don't tell anyone that you can see their shadows. If they hear you telling anyone, if you let them know that you know of them, they will get you.

GlyphGryph

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4465 on: June 11, 2013, 03:11:24 pm »

It would be dishonest to describe a system like that as justification "Web server Staging running off the Prod backend is normal and expected."

Since, you know, you appear to have several instances of staging that aren't hooked into production at all. If you have multiple staging environments then, yes, one could expect one of them to be hooked up to production, I think. Although I don't see why this couldn't be done just as effectively with a duplicate environment - I'm sure it's related to some specific business need of your company, but we have no real problems testing such things here in our non-production-hooked-up staging environments, and we are not exactly a small company, what with being the tool of choice for several of the worlds largest retailers.

If you have only one staging environment layer before production, then most people would not expect that to be hooked up to production.
« Last Edit: June 11, 2013, 03:14:29 pm by GlyphGryph »
Logged

olemars

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4466 on: June 11, 2013, 03:20:23 pm »

I think I'm glad I don't work with anything related to web development.
Logged

Mephisto

  • Bay Watcher
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4467 on: June 11, 2013, 03:44:05 pm »

If you have only one staging environment layer before production, then most people would not expect that to be hooked up to production.

This. We only have one staging environment. It would be nice to catch data-destroying bugs before they reach real user data.

We're a fairly small company - my team consists of seven other people currently. The process goes something like this:

  • Ticket owner tests locally and submits a pull request when he or she feels the code is adequate
  • Other devs pull down the branch and test
  • If no one has any complaints, a dev other than the owner merges the branch into master
  • Following our deployment schedule, the last x days of merged code is deployed to staging and tested
  • If staging passes, start the prod deployment, which deploys to all prod servers in series and halts on issues

I know some people blow a gasket if they don't have dedicated testers, but this works for us.

I think I'm glad I don't work with anything related to web development.

It's not all bad. It's not quite the type of development I went to college for and I thought I would hate it, but I find this domain to be genuinely interesting and my prior expectation of hate was dead wrong.
Logged

Mego

  • Bay Watcher
  • [PREFSTRING:MADNESS]
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4468 on: June 11, 2013, 03:49:15 pm »

Excellent overview of memory! But I'm still confused about parts of the article. It's these parts I was confused about:

Quote
This is the key point: the data storage that we call “process memory” is in my opinion best visualized as a massive file on disk.
...
...
...
But I much prefer to think of storage as being a chunk of disk storage, and physical memory being a smart caching mechanism that makes the disk look faster. Maybe I’m crazy, but that helps me understand it better.

Also, you didn't mention hard drive caches for whatever reason. I remember picking out a hard drive because it had a higher cache size than comparable drives. I have no idea how important it is, though. But I do remember testing it, and a ~500kB file that loaded in about 0.25 seconds the first time loaded in 0.001 seconds the second time. Or would memory caching come into play there? I have no idea.

I was talking about volatile data in my description - data that isn't intended to be permanent. But it also works with non-volatile data, too, like disk access. Data is fetched from the disk and loaded into main memory, so that a program needing to access data on the disk doesn't have to deal with the high access times for the disk. And hard drive caches work similarly to CPU caches. When the disk rotates so that a piece of the data you want is under the head, it is read and stored in the disk cache. When the disk cache is full and/or all data has been fetched, the data is sent from the disk cache to the CPU. The disk cache isn't cleared until new data is loaded in. So, the speedup in your example might be due to the disk cache, or it might be due to the data still being in main memory or the CPU cache, or maybe both.



The next category of memory is cache memory. Cache memory exists to speed up memory access by prefetching data from main memory, so that, when the data is needed, it can be read from the much-faster cache instead. Cache memory is less expensive than register memory, but is still pretty expensive compared to main memory.


Doesn't the stack also go in cache memory? The X86 chipset doesn't have a separate memory chip for the stack as far as I know, so it should either go in cache or in main memory and since the latter is slower that seems like an odd choice.

It depends on the size of the stack. Most Windows programs use a 1 MB stack, so it can easily fit in the CPU cache. In Linux, it's usually around 8 MB, so it depends on the size of the cache.

Siquo

  • Bay Watcher
  • Procedurally generated
    • View Profile
Re: if self.isCoder(): post() #Programming Thread
« Reply #4469 on: June 11, 2013, 04:27:55 pm »

We just code in production!  8)

Well, my predecessor did. We now do proper version control and always a last check with a pre-prod environment on the production backend.
Personally I'm nervous since we're throwing one of our new projects open source, and I've never done anything like that before.
Logged

This one thread is mine. MIIIIINE!!! And it will remain a happy, friendly, encouraging place, whether you lot like it or not. 
will rena,eme sique to sique sxds-- siquo if sucessufil
(cant spel siqou a. every speling looks wroing (hate this))
Pages: 1 ... 296 297 [298] 299 300 ... 796