You need to look at individual components and see what they're capable of. Your Motherboard is a limiting factor here, too. Some MB's don't allow you to overclock the CPU and RAM separately, which would be the end of most of your OC'ing endeavors right there. Check out your Bios and see what the options are.
I use
Core Temp to track my CPU temperatures. Generally you don't want to go over 70 C under load, and it's best to keep it somewhere around 50 or 60 C (Or lower, if you can manage it). Once you get into the 80 and 90 C range, you're going to be in danger of permanent damage, and while most CPU's will automatically shut down once they get too hot it's still not a good idea to push the limits. At best you'll limit the life time of your CPU, at worst you'll damage a component.
I highly, highly recommend staying away from voltages unless you've got disposable income for a new CPU. That's where you can damage your CPU the most, so just increase the clock until you find something stable and comfortable and step away. 4.0 GHz is already fast (I'm OC'd to 4.2 on what used to be a 3.6, but I've brought it up to 4.5). Check what other people have managed with your CPU model and go a few points lower. My CPU model's been clocked up to 5.0 GHz, but that was done by a professional who was just going for speed and with no regard for the lifespan of the chip.
RAM is, oddly, a bit trickier to OC than the CPU and it's an area I'm not too familiar with since the guides out there aren't incredibly thorough. As far as I can tell your sticks are designed to run at a certain speed - They can run slower, but they can't run an inch faster without losing system stability. Generally your MB will set the base clock rate on your sticks, and if they're capable of something faster you have to go in and do it manually.
For example, I've got a set of sticks that'll go up to 2333 MHz, but my MB initially set them at 1400 MHz. I brought the numbers up, but it turns out my MB only works in increments of 200 so I keep them at 2200 MHz. 2400 MHz just makes my entire system unstable. I'm not sure if that's the rule or just the limits of my MB and RAM, though, so if you want to push your RAM beyond what it was designed for you'll have to ask someone else.
You normally don't have to worry about temperatures on the RAM, and I'm not sure there's a lot of hardware that exists to track it. It's not really a component that overheats when it's working properly.
Be aware that OC'ing can potentially limit the life of your hardware. It's not uncommon to hear stories of people who OC'd their CPU and had it die a few months later. Even if it's a manufacturer's defect, the warranty is void the moment you change the clock rate. If you can't afford new hardware, don't risk it.