Sorry, I got my wires a bit crossed there; those programs I mentioned are for streaming, not actually recording. I'd say FRAPs is probably what most people use. To cut down on filesize you could try recording at half-size and with a lower framerate, but for the most part, you'd probably have to compress the raw data in ten minute segments (perfect for Youtube) before trying to record anything further, or you'll quickly run out of disk space.
As for FRAPs not picking up DOSBox, does DOSBox have any display settings you could tweak? Specifically, you might want to try looking for "hardware overlay", and see if you can disable it. Most screen capture softwares have a hard time recording anything that's using a hardware overlay, due to a whole bunch of boring and technical nonsense involving display layers and other stuff I don't fully comprehend; all I know is, I've wrestled with it in the past, and I hate it. HATE IT. But that's the only reason I can think of why FRAPs might not be playing nicely in that instance.
About the volume thing; I'm still on WinXP, so I'm not entirely sure how much Volume Control has changed in later Windoze. However, for me, I just go into Volume Control in Control Panel, select Properties, and switch to the Recording option. Then the sliders change to display all my recording inputs, like Line In, "What U Hear", Analog Mix, Mic, etc. I tick the box under "What U Hear", which allows software like FRAPs to pick up both game audio and mic audio whilst it records the video.