And then XKCD does a pretty good job of explaining generally how it worked:
Or in more technical detail, the code had a speed optimization which included some stuff related to pulling out a page of memory. Usual protocols for doing this involve pre-cleaning the page by just zeroing it out or similar. Instead of using typical methods for doing so, they had their own implementation which did not do so; likely the result of someone not being familiar with the reasons for doing so. In a low-level language, when you deallocate memory, the contents of that memory are not wiped, but rather 'undefined.' Which is to say, you're telling the OS that the chunk of memory in question can be used by other programs and such. The contents of that memory exist until either the OS wipes it (specifically to prevent security exploits like Heartbleed) or until it is otherwise altered. My understanding is that the heartbeat code explicitly wrote their own implementation for grabbing memory which did not wipe it. This makes sense if you aren't familiar with Heartbleed-like exploits, as pre-cleaning that memory does have a small time cost. And thus the newly allocated memory the program has access to has whatever data was recently in that particular place in memory.
And then you compound that with an issue related to bounds checking, as shown in that image above. So when you send the request, you would sent some sort of array of data, and essentially tell the server to repeat it back to you. You also tell the server how long that array of data is. If there is a mismatch, the server should detect that and simply ignore the request entirely, as it is malformed. Instead, the server would load that array into its memory allocated for the incoming packet, then attempt to read back as many characters as you had told it the message was. Which would include whatever data happened to be in that array of data. Since it wasn't wiped before being used for the array, that would include actual fragments of information from sources external to the request itself.
So in essence, it's the result of not following best practices in combination with a simple variant of the classic buffer overflow cracking method. If either allocation best-practices or request bounds checking had occurred, the exploit would not have existed.