I had the same problem until very recently when I learned that (unfortunately) armor and grinding, together, are the only way to become strong enough to survive in this game. You can find armor in keeps or dungeons or, if you're lucky, steal it off of bandits. Once you have it, you have to train armor using skill by finding a small animal somewhere, grabbing it using wrestling so it can't get away, and then lying on the ground and hitting . all day until the sun sets, so your dodging and armor/shield skills get up high (hopefully legendary). Then you have to pick up a thousand rocks and knap them into sharp rocks, then throw them on the ground to train your throwing skill. By the time you've done all this, your stats should be superhuman in several areas and even if you get in a fight with someone talented, you can either dodge it or block their hits with your armor.
And even with all this, it's still best to always be sneaking, never be visible, and take out most enemies with thrown sharpened rocks or bolts or arrows or sand or whatever is available before they even know you're there.
I've heard tell that others can survive without resorting to this, but I've never succeeded without this method. On the other hand, once you've got high stats and legendary skills, you can try attacking things (except for ranged fighters) directly and with interesting results without dying immediately, most of the time.