So does anybody have advice on how to build a manned rover?
Testing: If you can roll your vehicle on Kerbin, it'll be even easier to roll it on a low-gravity body. Most of my rovers feature roll cages and self-righting systems. If you use KAS and plan long-duration roving I recommend carrying spare parts.
My standard test regime (for vehicles with KF wheels) involves swerving across the runway trying to amplify oscillations, hard maneuvering while driving cross-grade on slopes, driving into the Administration Building pond, taking jumps off the launchpad and powering over
the inevitable results of previous tests obstacles. I've stopped doing much righting-system testing because something that's underpowered on Kerbin can still send the vehicle flying on the Mun.
Source: drove about 200 km on the Mun in a top-heavy giant rover; went up, down and across insane slopes at dangerous speeds, flew two servicing missions so far to replenish spare parts.
(I promise I only drive like a maniac in KSP!)
Delivery: Before I installed mods and started building 40-ton wheeled monstrosities, I was building lightweight stock rovers. After I failed to construct a working Sojourner-style deployment system, I tried the same rover delivery concept you're using, but I had a hard time getting that height compromise right with the fragile stock wheels and I've since switched to using skycranes. LoS boccacc's advice about subassemblies is good and sane, but for an insane rover delivery system I like my Harpy family of launch vehicles, where I drive the payload out to the pad and attach it with a claw.
I briefly flirted with making the rover and the lander be the same vehicle and I'm leaning that way again now that I'm finally planning a crewed interplanetary mission. On a whim I tried putting wheels on my Duna Ascent Vehicle prototype; a few hours later it had turned into something that looks like an IRBM transporter/erector/launcher.