Eehh... depends on the person. Chance of being hit and whathaveyou's never really bothered me (unless I've got passengers, anyway), ferex, so other drivers making a poor showing of it is... not that scary, really. Definitely makes things more worrying, because it makes your decision making that much more vital, but not exactly high on the terror scale.
It's the thought of a moment's inattention, or just plain bad luck/poor maintenance/poor situation, being enough to kill or maim someone that's always got to me the worst. Actually had a pretty close call a week or two ago... did pretty much everything right, on my end, and still nearly sideswiped a motorcyclist riding double, just due to the vagrancies of positioning. They'd managed to get on the road and in my blind spot in the space between me checking the first time before changing lanes and checking again. Couldn't hear 'em, couldn't see 'em, hadn't been in a position where I could have seen them. But they were there, and I came pretty close to probably killing or crippling 'em (at least partially because neither the driver or the person riding behind them were wearing the helmet orthe protective gear -- jacket, riding pants, at the least -- they should have been, but still). It's shit like that that drives most of the still continuing dislike/fear of driving, for me.
... but yeah, don't really have a choice about letting those sorts of concerns stop you. Do the best you can and hope. Most people make it without hurting anyone (themselves or otherwise) particularly seriously, so probably you will, too.