Randy so far seems the best for a nice difficult challenge, but I'm only on my first go with him and I know at any moment a whole bunch of bad things can happen at once. Still without the raid cycle being almost a sure thing (like Cassandra) I find there's more gaps where you can get stuff done or get good luck and expand your settler count.
I pretty much exclusively play on Randy Extreme, but I've starting tweaking the difficulty mid-game now, going up or down as needed to keep the game interesting. When I kept it on max, I'd be lucky to have five survivors at once before they all go berserk, die of plague, starve, freeze, or just generally end up with so many scars, bruises, cuts, cracks, and what have you that the overwhelming pain was too much for them to ever be a functional ember of society again.
One of my recent games had a guy with no nose, a missing left leg and right arm, recovering plague and several infections, fibrous mechanites, debuffs from all kinds of cannibalism, corpse-seeing, and dying friends/family/compatriots that I'm surprised he had time to eat between bouts of rage.
Now I start on extreme, wait until I'm about to give up, then drop it back to challenging until I have a handle on things. Seems like a good balance so far; there's still the chance of something bad happening, but it gives me time to build up a reasonable stockpile.